[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260927 19:36:31.797915 818 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.204.190:37763
I20260927 19:36:31.799335 818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 19:36:31.800448 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:36:31.824951 824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:36:31.824956 827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:36:31.825016 825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:36:31.825480 818 server_base.cc:1034] running on GCE node
I20260927 19:36:33.111742 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:36:33.111995 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:36:33.112151 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537793112132 us; error 0 us; skew 500 ppm
I20260927 19:36:33.112951 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:36:33.125151 818 webserver.cc:460] Webserver started at http://127.0.204.190:38103/ using document root <none> and password file <none>
I20260927 19:36:33.126171 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:36:33.126398 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:36:33.126811 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:36:33.133939 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/master-0-root/instance:
uuid: "8ffcb64138d74550a6ffff66e887f411"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.142943 818 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260927 19:36:33.148828 833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.150166 818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 19:36:33.150498 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/master-0-root
uuid: "8ffcb64138d74550a6ffff66e887f411"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.150816 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:36:33.190176 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:36:33.191646 818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 19:36:33.192190 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:36:33.277858 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.190:37763
I20260927 19:36:33.277984 884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.190:37763 every 8 connection(s)
I20260927 19:36:33.285910 885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:33.311688 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411: Bootstrap starting.
I20260927 19:36:33.318763 885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:33.321187 885 log.cc:824] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411: Log is configured to *not* fsync() on all Append() calls
I20260927 19:36:33.326550 885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411: No bootstrap required, opened a new log
I20260927 19:36:33.354642 885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffcb64138d74550a6ffff66e887f411" member_type: VOTER }
I20260927 19:36:33.355254 885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:33.355588 885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffcb64138d74550a6ffff66e887f411, State: Initialized, Role: FOLLOWER
I20260927 19:36:33.356563 885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffcb64138d74550a6ffff66e887f411" member_type: VOTER }
I20260927 19:36:33.357115 885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 19:36:33.357452 885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 19:36:33.357796 885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:33.362581 885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffcb64138d74550a6ffff66e887f411" member_type: VOTER }
I20260927 19:36:33.363512 885 leader_election.cc:302] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ffcb64138d74550a6ffff66e887f411; no voters:
I20260927 19:36:33.365631 885 leader_election.cc:288] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 19:36:33.366156 888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:33.370258 888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 1 LEADER]: Becoming Leader. State: Replica: 8ffcb64138d74550a6ffff66e887f411, State: Running, Role: LEADER
I20260927 19:36:33.371352 888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffcb64138d74550a6ffff66e887f411" member_type: VOTER }
I20260927 19:36:33.373096 885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 19:36:33.385540 889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ffcb64138d74550a6ffff66e887f411. Latest consensus state: current_term: 1 leader_uuid: "8ffcb64138d74550a6ffff66e887f411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffcb64138d74550a6ffff66e887f411" member_type: VOTER } }
I20260927 19:36:33.385919 890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ffcb64138d74550a6ffff66e887f411" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ffcb64138d74550a6ffff66e887f411" member_type: VOTER } }
I20260927 19:36:33.386852 889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [sys.catalog]: This master's current role is: LEADER
I20260927 19:36:33.387408 890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [sys.catalog]: This master's current role is: LEADER
I20260927 19:36:33.391523 897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 19:36:33.405117 897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 19:36:33.407600 818 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260927 19:36:33.424494 897 catalog_manager.cc:1347] Generated new cluster ID: 568b6a06aa634f28a237e924dfe1e0d1
I20260927 19:36:33.424794 897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 19:36:33.450340 897 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 19:36:33.452350 897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 19:36:33.468861 897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411: Generated new TSK 0
I20260927 19:36:33.470198 897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 19:36:33.539031 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:36:33.548748 907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:36:33.549996 908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:36:33.553896 818 server_base.cc:1034] running on GCE node
W20260927 19:36:33.554538 910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:36:33.555493 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:36:33.555791 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:36:33.555974 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537793555957 us; error 0 us; skew 500 ppm
I20260927 19:36:33.556550 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:36:33.561210 818 webserver.cc:460] Webserver started at http://127.0.204.129:42125/ using document root <none> and password file <none>
I20260927 19:36:33.561916 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:36:33.562155 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:36:33.562453 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:36:33.564222 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-0-root/instance:
uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.570941 818 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20260927 19:36:33.576136 915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.577049 818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260927 19:36:33.577414 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-0-root
uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.577733 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:36:33.599448 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:36:33.600832 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:36:33.604215 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:36:33.609872 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:36:33.610112 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.610460 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:36:33.610610 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.733476 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.129:40395
I20260927 19:36:33.733596 977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.129:40395 every 8 connection(s)
I20260927 19:36:33.742473 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:36:33.757089 982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:36:33.760107 983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:36:33.765125 985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:36:33.765213 818 server_base.cc:1034] running on GCE node
I20260927 19:36:33.766264 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:36:33.766520 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:36:33.766712 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537793766696 us; error 0 us; skew 500 ppm
I20260927 19:36:33.767333 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:36:33.770558 818 webserver.cc:460] Webserver started at http://127.0.204.130:43205/ using document root <none> and password file <none>
I20260927 19:36:33.771147 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:36:33.771378 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:36:33.771657 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:36:33.773201 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-1-root/instance:
uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.781775 818 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260927 19:36:33.785605 978 heartbeater.cc:347] Connected to a master server at 127.0.204.190:37763
I20260927 19:36:33.786111 978 heartbeater.cc:464] Registering TS with master...
I20260927 19:36:33.787679 990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.787871 978 heartbeater.cc:511] Master 127.0.204.190:37763 requested a full tablet report, sending...
I20260927 19:36:33.788720 818 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260927 19:36:33.788954 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-1-root
uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.789189 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:36:33.791491 850 ts_manager.cc:194] Registered new tserver with Master: 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395)
I20260927 19:36:33.796193 850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41986
I20260927 19:36:33.819693 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:36:33.821712 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:36:33.823941 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:36:33.826691 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:36:33.826962 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.827335 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:36:33.827555 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.937062 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.130:40583
I20260927 19:36:33.937211 1052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.130:40583 every 8 connection(s)
I20260927 19:36:33.946213 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:36:33.956169 1056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:36:33.957553 1057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:36:33.963017 1053 heartbeater.cc:347] Connected to a master server at 127.0.204.190:37763
I20260927 19:36:33.963567 1053 heartbeater.cc:464] Registering TS with master...
I20260927 19:36:33.964748 1053 heartbeater.cc:511] Master 127.0.204.190:37763 requested a full tablet report, sending...
I20260927 19:36:33.967136 850 ts_manager.cc:194] Registered new tserver with Master: 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:33.969164 850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41988
W20260927 19:36:33.969614 1059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:36:33.975962 818 server_base.cc:1034] running on GCE node
I20260927 19:36:33.977190 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:36:33.977497 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:36:33.977689 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537793977669 us; error 0 us; skew 500 ppm
I20260927 19:36:33.978178 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:36:33.981432 818 webserver.cc:460] Webserver started at http://127.0.204.131:36413/ using document root <none> and password file <none>
I20260927 19:36:33.981969 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:36:33.982199 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:36:33.982496 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:36:33.983639 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-2-root/instance:
uuid: "44c79fa05e27477da4ec1b913410f533"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.989182 818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 19:36:33.993119 1064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:33.994074 818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 19:36:33.994406 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-2-root
uuid: "44c79fa05e27477da4ec1b913410f533"
format_stamp: "Formatted at 2026-09-27 19:36:33 on dist-test-slave-zwk0"
I20260927 19:36:33.994767 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790537791628671-818-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:36:34.019469 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:36:34.021026 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:36:34.022981 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:36:34.026216 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:36:34.026490 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:34.026795 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:36:34.026999 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:36:34.127499 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.131:36069
I20260927 19:36:34.127615 1126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.131:36069 every 8 connection(s)
I20260927 19:36:34.148034 1127 heartbeater.cc:347] Connected to a master server at 127.0.204.190:37763
I20260927 19:36:34.148383 1127 heartbeater.cc:464] Registering TS with master...
I20260927 19:36:34.149102 1127 heartbeater.cc:511] Master 127.0.204.190:37763 requested a full tablet report, sending...
I20260927 19:36:34.151597 850 ts_manager.cc:194] Registered new tserver with Master: 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:34.152570 818 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019163279s
I20260927 19:36:34.153730 850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41990
I20260927 19:36:34.156229 818 test_util.cc:268] Using random seed: 288192166
I20260927 19:36:34.224450 850 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42000:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 19:36:34.228559 850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 19:36:34.360358 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 8501155ee43d4303b49dd91872fa6081 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260927 19:36:34.362596 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 86003e86d9224dd6a67bef7f95cc4881 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260927 19:36:34.361513 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 685c5d66a10549479f13281fcd7fbbc4 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260927 19:36:34.360939 1089 tablet_service.cc:1467] Processing CreateTablet for tablet 994ef439b09b46d3abcf439e645045e4 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260927 19:36:34.368916 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab1070fbc6e48bca66b39aa1a213162 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260927 19:36:34.365278 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c6f2925a8b52418c87249ef2c5743968 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260927 19:36:34.368970 1092 tablet_service.cc:1467] Processing CreateTablet for tablet a1bdd6d3afaf4de6ba4f3a0a9288b81c (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260927 19:36:34.373251 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 7be07c6c716147659d99a15bc41453d0 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260927 19:36:34.371760 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8501155ee43d4303b49dd91872fa6081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.378202 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 685c5d66a10549479f13281fcd7fbbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.380880 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 10dbe8ad420c4a2b97851e08f06d7d5d (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260927 19:36:34.382160 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86003e86d9224dd6a67bef7f95cc4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.384868 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1bdd6d3afaf4de6ba4f3a0a9288b81c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:36:34.396965 898 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:36:34.393611 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 685c5d66a10549479f13281fcd7fbbc4 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260927 19:36:34.395547 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f2925a8b52418c87249ef2c5743968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.392468 1018 tablet_service.cc:1467] Processing CreateTablet for tablet a1bdd6d3afaf4de6ba4f3a0a9288b81c (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260927 19:36:34.402267 1017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 685c5d66a10549479f13281fcd7fbbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.405254 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab1070fbc6e48bca66b39aa1a213162 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260927 19:36:34.404814 1089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994ef439b09b46d3abcf439e645045e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.409778 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be07c6c716147659d99a15bc41453d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.411562 1018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1bdd6d3afaf4de6ba4f3a0a9288b81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.411567 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 8501155ee43d4303b49dd91872fa6081 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260927 19:36:34.415215 1016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab1070fbc6e48bca66b39aa1a213162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.420776 1014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8501155ee43d4303b49dd91872fa6081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.410801 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 994ef439b09b46d3abcf439e645045e4 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260927 19:36:34.421386 1084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10dbe8ad420c4a2b97851e08f06d7d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.424921 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 7be07c6c716147659d99a15bc41453d0 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260927 19:36:34.432229 1013 tablet_service.cc:1467] Processing CreateTablet for tablet c6f2925a8b52418c87249ef2c5743968 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260927 19:36:34.436281 1015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994ef439b09b46d3abcf439e645045e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.436389 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 86003e86d9224dd6a67bef7f95cc4881 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260927 19:36:34.445708 1011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be07c6c716147659d99a15bc41453d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.442564 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab1070fbc6e48bca66b39aa1a213162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.448920 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 10dbe8ad420c4a2b97851e08f06d7d5d (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260927 19:36:34.452543 1012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86003e86d9224dd6a67bef7f95cc4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.455540 1010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10dbe8ad420c4a2b97851e08f06d7d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.456307 1013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f2925a8b52418c87249ef2c5743968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.529054 939 tablet_service.cc:1467] Processing CreateTablet for tablet 8501155ee43d4303b49dd91872fa6081 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260927 19:36:34.531710 939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8501155ee43d4303b49dd91872fa6081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.547632 940 tablet_service.cc:1467] Processing CreateTablet for tablet 994ef439b09b46d3abcf439e645045e4 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260927 19:36:34.553421 940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994ef439b09b46d3abcf439e645045e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.558852 943 tablet_service.cc:1467] Processing CreateTablet for tablet a1bdd6d3afaf4de6ba4f3a0a9288b81c (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260927 19:36:34.553287 941 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab1070fbc6e48bca66b39aa1a213162 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260927 19:36:34.565191 938 tablet_service.cc:1467] Processing CreateTablet for tablet c6f2925a8b52418c87249ef2c5743968 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260927 19:36:34.550746 942 tablet_service.cc:1467] Processing CreateTablet for tablet 685c5d66a10549479f13281fcd7fbbc4 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260927 19:36:34.581447 941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab1070fbc6e48bca66b39aa1a213162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.582273 937 tablet_service.cc:1467] Processing CreateTablet for tablet 86003e86d9224dd6a67bef7f95cc4881 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260927 19:36:34.584589 942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 685c5d66a10549479f13281fcd7fbbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.600744 936 tablet_service.cc:1467] Processing CreateTablet for tablet 7be07c6c716147659d99a15bc41453d0 (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260927 19:36:34.604102 935 tablet_service.cc:1467] Processing CreateTablet for tablet 10dbe8ad420c4a2b97851e08f06d7d5d (DEFAULT_TABLE table=test-workload [id=3d8d5d22abb24ee4949718ce6b01e750]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260927 19:36:34.604144 937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86003e86d9224dd6a67bef7f95cc4881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.610627 935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10dbe8ad420c4a2b97851e08f06d7d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.626695 936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be07c6c716147659d99a15bc41453d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.635888 938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f2925a8b52418c87249ef2c5743968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.650475 1146 tablet_bootstrap.cc:492] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:34.677065 943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1bdd6d3afaf4de6ba4f3a0a9288b81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:36:34.696290 1145 tablet_bootstrap.cc:492] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:34.729581 1147 tablet_bootstrap.cc:492] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:34.768826 1145 tablet_bootstrap.cc:654] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.777724 1147 tablet_bootstrap.cc:654] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.795842 1146 tablet_bootstrap.cc:654] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.798997 1145 tablet_bootstrap.cc:492] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:34.799593 1145 ts_tablet_manager.cc:1397] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.105s user 0.019s sys 0.007s
I20260927 19:36:34.801510 978 heartbeater.cc:503] Master 127.0.204.190:37763 was elected leader, sending a full tablet report...
I20260927 19:36:34.803833 1145 raft_consensus.cc:348] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.804926 1145 raft_consensus.cc:374] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.805294 1145 raft_consensus.cc:729] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:34.806420 1145 consensus_queue.cc:260] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.816622 1145 ts_tablet_manager.cc:1428] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.017s user 0.000s sys 0.019s
I20260927 19:36:34.817408 1053 heartbeater.cc:503] Master 127.0.204.190:37763 was elected leader, sending a full tablet report...
I20260927 19:36:34.822178 1151 raft_consensus.cc:482] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:34.823999 1145 tablet_bootstrap.cc:492] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:34.827250 1147 tablet_bootstrap.cc:492] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:34.828912 1147 ts_tablet_manager.cc:1397] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.105s user 0.016s sys 0.004s
I20260927 19:36:34.832463 1147 raft_consensus.cc:348] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.833279 1147 raft_consensus.cc:374] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.833596 1147 raft_consensus.cc:729] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:34.834264 1147 consensus_queue.cc:260] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.835086 1145 tablet_bootstrap.cc:654] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.823750 1151 raft_consensus.cc:504] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.840507 1151 leader_election.cc:288] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:34.844477 1146 tablet_bootstrap.cc:492] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:34.844931 1146 ts_tablet_manager.cc:1397] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.195s user 0.018s sys 0.017s
I20260927 19:36:34.852222 1147 ts_tablet_manager.cc:1428] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.023s user 0.004s sys 0.005s
I20260927 19:36:34.853261 1147 tablet_bootstrap.cc:492] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:34.853915 1145 tablet_bootstrap.cc:492] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:34.853467 1146 raft_consensus.cc:348] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.854221 1146 raft_consensus.cc:374] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.854350 1145 ts_tablet_manager.cc:1397] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.014s
I20260927 19:36:34.854503 1146 raft_consensus.cc:729] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:34.855281 1127 heartbeater.cc:503] Master 127.0.204.190:37763 was elected leader, sending a full tablet report...
I20260927 19:36:34.859130 1146 consensus_queue.cc:260] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.866510 1147 tablet_bootstrap.cc:654] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.874687 1145 raft_consensus.cc:348] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:34.875694 1145 raft_consensus.cc:374] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.876058 1145 raft_consensus.cc:729] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:34.877753 1145 consensus_queue.cc:260] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:34.889070 1145 ts_tablet_manager.cc:1428] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.034s user 0.000s sys 0.008s
W20260927 19:36:34.891482 1128 tablet.cc:2369] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 19:36:34.891770 1145 tablet_bootstrap.cc:492] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:34.884724 1146 ts_tablet_manager.cc:1428] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.039s user 0.007s sys 0.003s
I20260927 19:36:34.900089 1146 tablet_bootstrap.cc:492] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:34.926971 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dbe8ad420c4a2b97851e08f06d7d5d" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
W20260927 19:36:34.929688 992 leader_election.cc:341] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:34.930317 1147 tablet_bootstrap.cc:492] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:34.930787 1147 ts_tablet_manager.cc:1397] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.078s user 0.015s sys 0.026s
I20260927 19:36:34.934706 1146 tablet_bootstrap.cc:654] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.934760 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dbe8ad420c4a2b97851e08f06d7d5d" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:34.935271 1147 raft_consensus.cc:348] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.936003 1147 raft_consensus.cc:374] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.936269 1147 raft_consensus.cc:729] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
W20260927 19:36:34.936496 992 leader_election.cc:341] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:34.936920 1147 consensus_queue.cc:260] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.937110 1145 tablet_bootstrap.cc:654] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.937812 992 leader_election.cc:302] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542; no voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:34.938674 1151 raft_consensus.cc:2726] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:36:34.939787 1147 ts_tablet_manager.cc:1428] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260927 19:36:34.940665 1147 tablet_bootstrap.cc:492] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:34.950659 1145 tablet_bootstrap.cc:492] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:34.951157 1145 ts_tablet_manager.cc:1397] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.008s
I20260927 19:36:34.952412 1147 tablet_bootstrap.cc:654] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.954180 1145 raft_consensus.cc:348] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.954937 1145 raft_consensus.cc:374] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.955271 1146 tablet_bootstrap.cc:492] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:34.955236 1145 raft_consensus.cc:729] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:34.955797 1146 ts_tablet_manager.cc:1397] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.011s
I20260927 19:36:34.956323 1145 consensus_queue.cc:260] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.958979 1145 ts_tablet_manager.cc:1428] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260927 19:36:34.958753 1146 raft_consensus.cc:348] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.960072 1146 raft_consensus.cc:374] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:34.960371 1145 tablet_bootstrap.cc:492] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:34.960408 1146 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:34.961935 1146 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.964632 1146 ts_tablet_manager.cc:1428] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260927 19:36:34.965731 1146 tablet_bootstrap.cc:492] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:34.967239 1145 tablet_bootstrap.cc:654] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.969887 1151 raft_consensus.cc:482] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:34.970350 1151 raft_consensus.cc:504] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:34.972787 1151 leader_election.cc:288] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:34.973491 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:34.974802 1146 tablet_bootstrap.cc:654] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:34.974632 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:34.975288 1102 raft_consensus.cc:2454] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 0.
I20260927 19:36:34.976730 992 leader_election.cc:302] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
W20260927 19:36:34.979614 992 leader_election.cc:341] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:34.980875 1151 raft_consensus.cc:2781] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:34.981429 1151 raft_consensus.cc:482] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:34.981935 1151 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:34.987185 1157 raft_consensus.cc:482] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:34.987742 1157 raft_consensus.cc:504] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.000568 1157 leader_election.cc:288] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.010536 1151 raft_consensus.cc:504] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.015081 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:35.016546 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
W20260927 19:36:35.017735 992 leader_election.cc:341] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.017951 1102 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.021807 1151 leader_election.cc:288] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.025931 1145 tablet_bootstrap.cc:492] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:35.026470 1145 ts_tablet_manager.cc:1397] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.013s
I20260927 19:36:35.025933 1147 tablet_bootstrap.cc:492] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.029260 1147 ts_tablet_manager.cc:1397] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.089s user 0.026s sys 0.023s
I20260927 19:36:35.029785 1102 raft_consensus.cc:2454] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 1.
I20260927 19:36:35.031213 992 leader_election.cc:302] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters: 1825fc1c3aa64ba5a098b1d7c734e59c
I20260927 19:36:35.032042 1151 raft_consensus.cc:2781] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.032516 1151 raft_consensus.cc:686] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:35.032382 1147 raft_consensus.cc:348] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.033655 1147 raft_consensus.cc:374] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.033291 1145 raft_consensus.cc:348] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.033906 1151 consensus_queue.cc:237] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.034168 1145 raft_consensus.cc:374] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.033967 1147 raft_consensus.cc:729] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.035390 1145 raft_consensus.cc:729] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.036357 1145 consensus_queue.cc:260] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.036568 1147 consensus_queue.cc:260] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.040087 1147 ts_tablet_manager.cc:1428] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.010s user 0.001s sys 0.005s
I20260927 19:36:35.031781 1146 tablet_bootstrap.cc:492] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.040802 1146 ts_tablet_manager.cc:1397] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.045s
I20260927 19:36:35.041273 1147 tablet_bootstrap.cc:492] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:35.044216 1146 raft_consensus.cc:348] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.045398 1146 raft_consensus.cc:374] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.046406 1146 raft_consensus.cc:729] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.050132 1147 tablet_bootstrap.cc:654] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.051784 1146 consensus_queue.cc:260] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.054108 1146 ts_tablet_manager.cc:1428] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260927 19:36:35.058009 1145 ts_tablet_manager.cc:1428] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260927 19:36:35.059631 1145 tablet_bootstrap.cc:492] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:35.059741 1146 tablet_bootstrap.cc:492] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:35.071816 1146 tablet_bootstrap.cc:654] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.075512 850 catalog_manager.cc:5495] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:35.085770 1145 tablet_bootstrap.cc:654] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.093003 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.093761 1102 raft_consensus.cc:2454] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 0.
I20260927 19:36:35.093904 1151 raft_consensus.cc:482] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.094427 1151 raft_consensus.cc:504] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.095369 917 leader_election.cc:302] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.096539 1151 leader_election.cc:288] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069), 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395)
I20260927 19:36:35.096606 1157 raft_consensus.cc:2781] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.097049 1157 raft_consensus.cc:482] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.097410 1157 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.097524 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.098897 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
W20260927 19:36:35.100438 992 leader_election.cc:341] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:36:35.101610 992 leader_election.cc:341] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.101979 992 leader_election.cc:302] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542; no voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:35.103278 1151 raft_consensus.cc:2726] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:36:35.105839 1147 tablet_bootstrap.cc:492] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.108858 1147 ts_tablet_manager.cc:1397] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.012s
I20260927 19:36:35.115906 1147 raft_consensus.cc:348] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.117089 1147 raft_consensus.cc:374] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.117568 1147 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.119465 1146 tablet_bootstrap.cc:492] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.120143 1146 ts_tablet_manager.cc:1397] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.001s
I20260927 19:36:35.126868 1157 raft_consensus.cc:504] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.123162 1147 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.127676 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.131872 1157 leader_election.cc:288] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
W20260927 19:36:35.133466 918 leader_election.cc:341] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.133936 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.134611 1147 ts_tablet_manager.cc:1428] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260927 19:36:35.135005 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
W20260927 19:36:35.135228 918 leader_election.cc:341] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.135555 1102 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.135664 1147 tablet_bootstrap.cc:492] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:35.135766 1146 raft_consensus.cc:348] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.136831 1146 raft_consensus.cc:374] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.137108 1146 raft_consensus.cc:729] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.137816 1146 consensus_queue.cc:260] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.138329 1145 tablet_bootstrap.cc:492] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:35.138795 1145 ts_tablet_manager.cc:1397] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.013s
I20260927 19:36:35.140812 1146 ts_tablet_manager.cc:1428] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260927 19:36:35.142416 1146 tablet_bootstrap.cc:492] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:35.145283 1145 raft_consensus.cc:348] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.146346 1157 raft_consensus.cc:482] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.146795 1157 raft_consensus.cc:504] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.147219 1102 raft_consensus.cc:2454] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 1.
I20260927 19:36:35.148802 1157 leader_election.cc:288] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.148947 917 leader_election.cc:302] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters: 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:35.150950 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.153600 1157 raft_consensus.cc:2781] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.151221 1145 raft_consensus.cc:374] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260927 19:36:35.155355 918 leader_election.cc:341] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.155810 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.155624 1145 raft_consensus.cc:729] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.156561 1102 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 0.
I20260927 19:36:35.155596 1147 tablet_bootstrap.cc:654] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.159669 917 leader_election.cc:302] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters: 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:35.160308 1146 tablet_bootstrap.cc:654] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.164175 1174 raft_consensus.cc:2781] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.165040 1174 raft_consensus.cc:482] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.165436 1174 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.158501 1145 consensus_queue.cc:260] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.162308 1157 raft_consensus.cc:686] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:35.174407 1157 consensus_queue.cc:237] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.176510 1174 raft_consensus.cc:504] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.181306 1145 ts_tablet_manager.cc:1428] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.042s user 0.011s sys 0.000s
I20260927 19:36:35.183732 1145 tablet_bootstrap.cc:492] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:35.188926 1174 leader_election.cc:288] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.188951 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.191026 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:35.191974 1102 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:36:35.192140 918 leader_election.cc:341] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.199033 1147 tablet_bootstrap.cc:492] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.199890 1147 ts_tablet_manager.cc:1397] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.023s
I20260927 19:36:35.202711 1102 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 1.
I20260927 19:36:35.203518 1146 tablet_bootstrap.cc:492] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.204134 917 leader_election.cc:302] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters: 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:35.204741 1146 ts_tablet_manager.cc:1397] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.014s
I20260927 19:36:35.205164 1157 raft_consensus.cc:2781] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.206795 1157 raft_consensus.cc:686] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:35.207926 1157 consensus_queue.cc:237] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.209659 1146 raft_consensus.cc:348] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.210279 1146 raft_consensus.cc:374] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.210530 1146 raft_consensus.cc:729] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.211215 1146 consensus_queue.cc:260] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.212368 1145 tablet_bootstrap.cc:654] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.213614 1146 ts_tablet_manager.cc:1428] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260927 19:36:35.213182 1147 raft_consensus.cc:348] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.214501 1147 raft_consensus.cc:374] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.214850 1147 raft_consensus.cc:729] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.215147 1146 tablet_bootstrap.cc:492] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:35.215656 1147 consensus_queue.cc:260] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.236981 1147 ts_tablet_manager.cc:1428] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.036s user 0.005s sys 0.004s
I20260927 19:36:35.239900 1147 tablet_bootstrap.cc:492] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:35.253845 1153 raft_consensus.cc:482] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.254844 1153 raft_consensus.cc:504] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.259065 1146 tablet_bootstrap.cc:654] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.260385 1145 tablet_bootstrap.cc:492] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:35.261425 1145 ts_tablet_manager.cc:1397] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.020s
I20260927 19:36:35.265553 1153 leader_election.cc:288] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.281241 848 catalog_manager.cc:5495] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 1 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:35.284775 848 catalog_manager.cc:5495] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 1 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:35.287328 1151 raft_consensus.cc:482] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.290345 1151 raft_consensus.cc:504] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.277845 1145 raft_consensus.cc:348] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.291548 1145 raft_consensus.cc:374] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.291924 1145 raft_consensus.cc:729] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.292939 1151 leader_election.cc:288] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.294819 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:35.297704 953 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 0.
I20260927 19:36:35.295250 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.298959 992 leader_election.cc:302] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
W20260927 19:36:35.299901 992 leader_election.cc:341] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.300993 1151 raft_consensus.cc:2781] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.301534 1151 raft_consensus.cc:482] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.302022 1151 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.310050 1145 consensus_queue.cc:260] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.320220 1145 ts_tablet_manager.cc:1428] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.058s user 0.009s sys 0.001s
I20260927 19:36:35.312292 1147 tablet_bootstrap.cc:654] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.323127 1153 raft_consensus.cc:482] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.322039 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:35.320995 1151 raft_consensus.cc:504] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.323773 1153 raft_consensus.cc:504] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.324836 953 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 0.
I20260927 19:36:35.327819 1066 leader_election.cc:302] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.327917 1153 leader_election.cc:288] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.329308 1153 raft_consensus.cc:2781] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.330034 1153 raft_consensus.cc:482] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.331375 1145 tablet_bootstrap.cc:492] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:35.329311 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.331858 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dbe8ad420c4a2b97851e08f06d7d5d" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.333254 1027 raft_consensus.cc:2454] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 0.
I20260927 19:36:35.330394 1153 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.334028 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:35.334564 953 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.333074 1028 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 0.
I20260927 19:36:35.338213 1067 leader_election.cc:302] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.341351 1146 tablet_bootstrap.cc:492] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.342459 1146 ts_tablet_manager.cc:1397] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.128s user 0.024s sys 0.027s
I20260927 19:36:35.344184 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dbe8ad420c4a2b97851e08f06d7d5d" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:35.344663 1185 raft_consensus.cc:2781] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.348610 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:35.346792 1151 leader_election.cc:288] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.350922 953 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 1.
W20260927 19:36:35.351588 992 leader_election.cc:341] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:36:35.351024 1146 raft_consensus.cc:348] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.352180 1146 raft_consensus.cc:374] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.352340 992 leader_election.cc:302] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters: 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:35.353163 1185 raft_consensus.cc:482] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.352583 1146 raft_consensus.cc:729] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.355453 1185 raft_consensus.cc:3037] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.356452 1151 raft_consensus.cc:2781] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.356006 1146 consensus_queue.cc:260] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.360482 952 raft_consensus.cc:2454] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 0.
I20260927 19:36:35.361804 1151 raft_consensus.cc:686] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:35.363015 1151 consensus_queue.cc:237] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.365418 1146 ts_tablet_manager.cc:1428] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260927 19:36:35.365818 1145 tablet_bootstrap.cc:654] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.357795 1153 raft_consensus.cc:504] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.366576 1146 tablet_bootstrap.cc:492] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:35.370074 1153 leader_election.cc:288] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.372326 1147 tablet_bootstrap.cc:492] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.372537 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.372844 1147 ts_tablet_manager.cc:1397] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.134s user 0.018s sys 0.000s
I20260927 19:36:35.373176 1028 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.376354 1147 raft_consensus.cc:348] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.377254 1147 raft_consensus.cc:374] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.377615 1147 raft_consensus.cc:729] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.377997 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:35.378644 952 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.379318 1146 tablet_bootstrap.cc:654] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.379333 1147 consensus_queue.cc:260] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.383404 1147 ts_tablet_manager.cc:1428] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260927 19:36:35.386647 848 catalog_manager.cc:5495] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:35.404404 1028 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 1.
I20260927 19:36:35.405237 1146 tablet_bootstrap.cc:492] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.414698 1146 ts_tablet_manager.cc:1397] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.048s user 0.015s sys 0.010s
W20260927 19:36:35.408185 898 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:36:35.408015 952 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 1.
I20260927 19:36:35.414670 1067 leader_election.cc:302] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.415963 1153 raft_consensus.cc:2781] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.408779 1147 tablet_bootstrap.cc:492] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:35.419191 1146 raft_consensus.cc:348] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.420688 1146 raft_consensus.cc:374] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.423629 1157 raft_consensus.cc:482] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.424146 1157 raft_consensus.cc:504] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.426139 1157 leader_election.cc:288] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.427235 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.428339 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.430554 1102 raft_consensus.cc:2379] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1825fc1c3aa64ba5a098b1d7c734e59c in current term 1: Already voted for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in this term.
I20260927 19:36:35.432251 917 leader_election.cc:302] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c; no voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:35.421140 1146 raft_consensus.cc:729] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.433792 1157 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.436772 1146 consensus_queue.cc:260] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.438808 1145 tablet_bootstrap.cc:492] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:35.439268 1146 ts_tablet_manager.cc:1428] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.024s user 0.006s sys 0.001s
I20260927 19:36:35.439502 1145 ts_tablet_manager.cc:1397] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.109s user 0.025s sys 0.016s
I20260927 19:36:35.440233 1146 tablet_bootstrap.cc:492] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:35.426085 1147 tablet_bootstrap.cc:654] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.428818 1185 raft_consensus.cc:504] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.449790 1185 leader_election.cc:288] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.451419 1153 raft_consensus.cc:686] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Becoming Leader. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:35.452535 1153 consensus_queue.cc:237] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.454391 1145 raft_consensus.cc:348] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.454687 1151 consensus_queue.cc:1035] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:35.455261 1145 raft_consensus.cc:374] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.455281 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dbe8ad420c4a2b97851e08f06d7d5d" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.455699 1145 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.456058 1028 raft_consensus.cc:3037] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.456601 1145 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.465741 1157 raft_consensus.cc:2726] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:36:35.466650 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10dbe8ad420c4a2b97851e08f06d7d5d" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:35.469601 952 raft_consensus.cc:3037] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.469672 1145 ts_tablet_manager.cc:1428] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.030s user 0.004s sys 0.008s
I20260927 19:36:35.475090 1145 tablet_bootstrap.cc:492] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:35.479703 1146 tablet_bootstrap.cc:654] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.485901 1028 raft_consensus.cc:2454] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 1.
I20260927 19:36:35.491451 1067 leader_election.cc:302] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.493109 1190 raft_consensus.cc:2781] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.493862 1190 raft_consensus.cc:686] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Becoming Leader. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:35.495051 1190 consensus_queue.cc:237] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.499974 952 raft_consensus.cc:2454] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 1.
I20260927 19:36:35.488032 1145 tablet_bootstrap.cc:654] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.513059 847 catalog_manager.cc:5495] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c79fa05e27477da4ec1b913410f533 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "44c79fa05e27477da4ec1b913410f533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260927 19:36:35.523450 1147 tablet_bootstrap.cc:492] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.524288 1146 tablet_bootstrap.cc:492] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.525197 1146 ts_tablet_manager.cc:1397] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.085s user 0.012s sys 0.016s
I20260927 19:36:35.528683 1151 consensus_queue.cc:1035] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:35.529940 1145 tablet_bootstrap.cc:492] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:35.530436 1145 ts_tablet_manager.cc:1397] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.006s
I20260927 19:36:35.528275 1147 ts_tablet_manager.cc:1397] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.120s user 0.015s sys 0.015s
I20260927 19:36:35.533876 1145 raft_consensus.cc:348] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.534726 1145 raft_consensus.cc:374] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.535010 1145 raft_consensus.cc:729] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.535764 1145 consensus_queue.cc:260] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.538313 1147 raft_consensus.cc:348] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.539160 1147 raft_consensus.cc:374] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.539431 1147 raft_consensus.cc:729] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.541734 1145 ts_tablet_manager.cc:1428] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260927 19:36:35.540122 1147 consensus_queue.cc:260] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.542635 1145 tablet_bootstrap.cc:492] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: Bootstrap starting.
I20260927 19:36:35.527968 1146 raft_consensus.cc:348] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.543562 1146 raft_consensus.cc:374] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.543946 1146 raft_consensus.cc:729] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.544674 1146 consensus_queue.cc:260] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.546847 1146 ts_tablet_manager.cc:1428] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260927 19:36:35.548647 1146 tablet_bootstrap.cc:492] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: Bootstrap starting.
I20260927 19:36:35.560063 1147 ts_tablet_manager.cc:1428] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20260927 19:36:35.563079 1147 tablet_bootstrap.cc:492] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:35.567032 1145 tablet_bootstrap.cc:654] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.572276 848 catalog_manager.cc:5495] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c79fa05e27477da4ec1b913410f533 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "44c79fa05e27477da4ec1b913410f533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260927 19:36:35.577701 1146 tablet_bootstrap.cc:654] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.590333 1146 tablet_bootstrap.cc:492] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: No bootstrap required, opened a new log
I20260927 19:36:35.590852 1146 ts_tablet_manager.cc:1397] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.004s
I20260927 19:36:35.593583 1146 raft_consensus.cc:348] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.594403 1146 raft_consensus.cc:374] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.594704 1146 raft_consensus.cc:729] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.595543 1146 consensus_queue.cc:260] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.598469 1146 ts_tablet_manager.cc:1428] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:36:35.611961 1147 tablet_bootstrap.cc:654] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.617537 1028 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.624184 1145 tablet_bootstrap.cc:492] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: No bootstrap required, opened a new log
I20260927 19:36:35.624692 1145 ts_tablet_manager.cc:1397] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.027s
I20260927 19:36:35.628453 1145 raft_consensus.cc:348] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.629563 1145 raft_consensus.cc:374] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.629318 1157 consensus_queue.cc:1035] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:35.629889 1145 raft_consensus.cc:729] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.630848 1145 consensus_queue.cc:260] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.635404 1027 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.641744 1145 ts_tablet_manager.cc:1428] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: Time spent starting tablet: real 0.017s user 0.005s sys 0.005s
I20260927 19:36:35.646595 1147 tablet_bootstrap.cc:492] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.647159 1147 ts_tablet_manager.cc:1397] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.008s
I20260927 19:36:35.648772 1157 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:35.650864 1147 raft_consensus.cc:348] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.651732 1147 raft_consensus.cc:374] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.652019 1147 raft_consensus.cc:729] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.652647 1147 consensus_queue.cc:260] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.654924 1147 ts_tablet_manager.cc:1428] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260927 19:36:35.658133 1147 tablet_bootstrap.cc:492] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533: Bootstrap starting.
I20260927 19:36:35.664424 1174 raft_consensus.cc:482] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.664786 1147 tablet_bootstrap.cc:654] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533: Neither blocks nor log segments found. Creating new log.
I20260927 19:36:35.665022 1174 raft_consensus.cc:504] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.674481 1151 raft_consensus.cc:482] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.675209 1151 raft_consensus.cc:504] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.680231 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:35.681243 1157 raft_consensus.cc:482] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.682083 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.682446 1157 raft_consensus.cc:504] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.682936 1028 raft_consensus.cc:2454] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 0.
I20260927 19:36:35.683329 1190 raft_consensus.cc:482] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.683957 1190 raft_consensus.cc:504] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.686690 1190 leader_election.cc:288] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.686862 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.687610 1101 raft_consensus.cc:2454] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 0.
I20260927 19:36:35.688503 1174 leader_election.cc:288] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.689349 1157 leader_election.cc:288] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.689857 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.691285 1028 raft_consensus.cc:2454] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 0.
I20260927 19:36:35.693542 1067 leader_election.cc:302] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.681244 952 raft_consensus.cc:2454] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 0.
I20260927 19:36:35.699594 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.700598 1027 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 0.
I20260927 19:36:35.706758 917 leader_election.cc:302] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:35.707116 992 leader_election.cc:302] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:35.707962 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.708673 1101 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 0.
I20260927 19:36:35.709012 1199 raft_consensus.cc:2781] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.709537 1199 raft_consensus.cc:482] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.709898 1199 raft_consensus.cc:3037] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.710247 1174 raft_consensus.cc:2781] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.710629 1174 raft_consensus.cc:482] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.710963 1174 raft_consensus.cc:3037] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.713904 918 leader_election.cc:302] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:35.714752 1157 raft_consensus.cc:2781] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.715094 1157 raft_consensus.cc:482] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.715494 1151 leader_election.cc:288] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.717247 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533" is_pre_election: true
I20260927 19:36:35.717530 1190 raft_consensus.cc:2781] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:36:35.717904 1190 raft_consensus.cc:482] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:36:35.718261 1190 raft_consensus.cc:3037] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.718677 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:35.719717 952 raft_consensus.cc:2454] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 0.
I20260927 19:36:35.715420 1157 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.726291 1147 tablet_bootstrap.cc:492] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533: No bootstrap required, opened a new log
I20260927 19:36:35.726745 1147 ts_tablet_manager.cc:1397] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.040s
I20260927 19:36:35.730516 1147 raft_consensus.cc:348] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.732292 1147 raft_consensus.cc:374] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:36:35.732622 1147 raft_consensus.cc:729] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Initialized, Role: FOLLOWER
I20260927 19:36:35.733150 1147 consensus_queue.cc:260] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.736560 1147 ts_tablet_manager.cc:1428] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 19:36:35.730479 1199 raft_consensus.cc:504] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.739526 1190 raft_consensus.cc:504] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.729423 1174 raft_consensus.cc:504] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.743172 1101 raft_consensus.cc:2379] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d0b4f3ae70e4858a14b92f93ecda542 in current term 1: Already voted for candidate 44c79fa05e27477da4ec1b913410f533 in this term.
I20260927 19:36:35.743768 1157 raft_consensus.cc:504] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.750174 1190 leader_election.cc:288] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.753314 1028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.754596 1157 leader_election.cc:288] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Requested vote from peers 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.754786 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:35.755574 1101 raft_consensus.cc:2379] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2d0b4f3ae70e4858a14b92f93ecda542 in current term 1: Already voted for candidate 44c79fa05e27477da4ec1b913410f533 in this term.
I20260927 19:36:35.754454 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:35.756129 952 raft_consensus.cc:3037] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.756933 1199 leader_election.cc:288] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.758901 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.759670 1174 leader_election.cc:288] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:35.759734 1027 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.763404 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:35.763995 1101 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.764542 952 raft_consensus.cc:2454] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 1.
I20260927 19:36:35.766472 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:35.766847 1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:35.768385 953 raft_consensus.cc:2379] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 44c79fa05e27477da4ec1b913410f533 in current term 1: Already voted for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in this term.
I20260927 19:36:35.768896 1026 raft_consensus.cc:3037] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.768971 992 leader_election.cc:302] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters: 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:35.767683 1028 raft_consensus.cc:2379] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 44c79fa05e27477da4ec1b913410f533 in current term 1: Already voted for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in this term.
I20260927 19:36:35.771184 1210 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:36:35.773051 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:35.773921 1102 raft_consensus.cc:3037] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.753392 1176 consensus_queue.cc:1035] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260927 19:36:35.779340 1067 leader_election.cc:302] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44c79fa05e27477da4ec1b913410f533; no voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:35.779783 1026 raft_consensus.cc:2454] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 1.
I20260927 19:36:35.775509 1199 raft_consensus.cc:2781] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.781030 1199 raft_consensus.cc:686] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:35.781448 918 leader_election.cc:302] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:35.782673 1176 raft_consensus.cc:2781] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.782339 1199 consensus_queue.cc:237] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.784061 1176 raft_consensus.cc:686] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:35.784420 1102 raft_consensus.cc:2454] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 1.
I20260927 19:36:35.785203 1190 raft_consensus.cc:2726] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 19:36:35.785418 1176 consensus_queue.cc:237] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.794061 1190 raft_consensus.cc:482] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:36:35.794557 1190 raft_consensus.cc:504] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:35.797152 1190 leader_election.cc:288] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:35.798255 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" is_pre_election: true
I20260927 19:36:35.799371 953 raft_consensus.cc:2379] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44c79fa05e27477da4ec1b913410f533 in current term 1: Already voted for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in this term.
I20260927 19:36:35.800411 1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" is_pre_election: true
I20260927 19:36:35.803025 1067 leader_election.cc:302] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44c79fa05e27477da4ec1b913410f533; no voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:35.804687 1190 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:36:35.785441 1027 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 1.
I20260927 19:36:35.809268 918 leader_election.cc:302] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:35.810163 1210 raft_consensus.cc:2781] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:36:35.810631 1210 raft_consensus.cc:686] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:35.811681 1210 consensus_queue.cc:237] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:35.812220 1101 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 1.
I20260927 19:36:35.814142 1190 raft_consensus.cc:2726] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:36:35.837229 849 catalog_manager.cc:5495] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:35.840176 1151 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:35.867633 850 catalog_manager.cc:5495] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 1 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:35.872655 1152 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:35.883846 1190 consensus_queue.cc:1035] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:35.884192 1185 consensus_queue.cc:1035] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:35.885025 848 catalog_manager.cc:5495] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 1 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } }
I20260927 19:36:35.916973 1185 consensus_queue.cc:1035] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:35.958102 1185 consensus_queue.cc:1035] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:36.208127 1152 consensus_queue.cc:1035] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:36.237010 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:56330
I20260927 19:36:36.237752 1027 raft_consensus.cc:595] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Received request to transfer leadership to TS 1825fc1c3aa64ba5a098b1d7c734e59c
I20260927 19:36:36.242427 1199 consensus_queue.cc:1035] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:36.247466 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:56330
I20260927 19:36:36.248169 1027 raft_consensus.cc:595] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Received request to transfer leadership to TS 1825fc1c3aa64ba5a098b1d7c734e59c
I20260927 19:36:36.253865 1174 consensus_queue.cc:1035] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:36.276535 1174 consensus_queue.cc:1035] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:36.305686 1102 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:41006
I20260927 19:36:36.306360 1102 raft_consensus.cc:595] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:36.309149 1174 consensus_queue.cc:1035] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:36:36.333559 1174 consensus_queue.cc:1035] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:36:36.348291 953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:40922
I20260927 19:36:36.348927 953 raft_consensus.cc:595] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:36.365991 953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:40922
I20260927 19:36:36.366868 953 raft_consensus.cc:595] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:36.373121 1190 raft_consensus.cc:979] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:36.373670 1185 raft_consensus.cc:1067] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:36.376148 1026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56328
I20260927 19:36:36.376850 1026 raft_consensus.cc:482] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:36.377223 1026 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.382426 1026 raft_consensus.cc:504] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.384783 1026 leader_election.cc:288] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:36.385694 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:36.386147 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:36.386361 953 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.386787 1102 raft_consensus.cc:3032] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:36.387131 1102 raft_consensus.cc:729] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:36.387912 1102 consensus_queue.cc:260] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.388994 1102 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.394479 1102 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:36.394950 953 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:36.395568 992 leader_election.cc:302] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:36.396412 1199 raft_consensus.cc:2781] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:36.396878 1199 raft_consensus.cc:686] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:36.397737 1199 consensus_queue.cc:237] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.405920 849 catalog_manager.cc:5495] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 1 to 2, leader changed from 44c79fa05e27477da4ec1b913410f533 (127.0.204.131) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:36.422345 1152 raft_consensus.cc:979] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:36.422814 1152 raft_consensus.cc:1067] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Signalling peer 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:36.426574 953 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:40902
I20260927 19:36:36.427211 953 raft_consensus.cc:482] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:36.427608 953 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.436504 953 raft_consensus.cc:504] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.439410 1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:36.439442 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:36.440218 1026 raft_consensus.cc:3032] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:36.440270 1102 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.440707 1026 raft_consensus.cc:729] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:36.441578 1026 consensus_queue.cc:260] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.442694 1026 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.440053 953 leader_election.cc:288] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 2 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:36.446990 1102 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 2.
I20260927 19:36:36.448184 917 leader_election.cc:302] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:36.448994 1174 raft_consensus.cc:2781] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:36.449465 1174 raft_consensus.cc:686] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:36.449656 1026 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 2.
I20260927 19:36:36.450363 1174 consensus_queue.cc:237] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.457324 849 catalog_manager.cc:5495] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 1 to 2, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 2 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:36.477239 1152 raft_consensus.cc:979] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:36.477756 1152 raft_consensus.cc:1067] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Signalling peer 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:36.479414 953 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:40902
I20260927 19:36:36.480005 953 raft_consensus.cc:482] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:36.480319 953 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.486258 953 raft_consensus.cc:504] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.488394 953 leader_election.cc:288] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 2 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:36.489513 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:36.489447 1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:36.490430 1102 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.490481 1026 raft_consensus.cc:3032] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:36.490881 1026 raft_consensus.cc:729] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:36.491641 1026 consensus_queue.cc:260] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.492837 1026 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.495885 1102 raft_consensus.cc:2454] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 2.
I20260927 19:36:36.497148 917 leader_election.cc:302] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:36.498009 1174 raft_consensus.cc:2781] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:36.498428 1174 raft_consensus.cc:686] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:36.499174 1174 consensus_queue.cc:237] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.500432 1026 raft_consensus.cc:2454] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 2.
I20260927 19:36:36.506446 849 catalog_manager.cc:5495] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 1 to 2, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 2 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:36.715430 1210 raft_consensus.cc:979] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:36.716549 1210 raft_consensus.cc:1067] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:36.720098 1026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56320
I20260927 19:36:36.720979 1026 raft_consensus.cc:482] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:36.721467 1026 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.727658 1026 raft_consensus.cc:504] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.729664 1026 leader_election.cc:288] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:36.730764 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:36.730955 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:36.731518 953 raft_consensus.cc:3032] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:36.731564 1102 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.732263 953 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:36.733664 953 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.734792 953 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.738931 1102 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:36.740110 992 leader_election.cc:302] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:36.743538 1176 raft_consensus.cc:979] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:36.743978 1152 raft_consensus.cc:2781] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:36.744592 1152 raft_consensus.cc:686] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:36.744772 953 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:36.744019 1176 raft_consensus.cc:1067] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:36.745383 1152 consensus_queue.cc:237] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.748494 1026 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56320
I20260927 19:36:36.749128 1026 raft_consensus.cc:482] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:36.749537 1026 raft_consensus.cc:3037] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.757236 1026 raft_consensus.cc:504] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.756652 849 catalog_manager.cc:5495] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 1 to 2, leader changed from 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:36.759284 1026 leader_election.cc:288] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:36.760673 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:36.761296 953 raft_consensus.cc:3032] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:36.761683 953 raft_consensus.cc:729] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:36.760664 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be07c6c716147659d99a15bc41453d0" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:36.762447 1102 raft_consensus.cc:3037] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.762389 953 consensus_queue.cc:260] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.763660 953 raft_consensus.cc:3037] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:36.770951 1102 raft_consensus.cc:2454] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:36.772279 992 leader_election.cc:302] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:36.772312 953 raft_consensus.cc:2454] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:36.773176 1152 raft_consensus.cc:2781] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:36.773818 1152 raft_consensus.cc:686] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:36.774971 1152 consensus_queue.cc:237] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:36.781864 849 catalog_manager.cc:5495] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 1 to 2, leader changed from 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:36.819752 953 raft_consensus.cc:1261] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:36.821009 1152 consensus_queue.cc:1035] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:36.828739 1102 raft_consensus.cc:1261] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:36.830535 1199 consensus_queue.cc:1035] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:36:36.870056 1026 raft_consensus.cc:1261] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:36.870656 1102 raft_consensus.cc:1261] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:36.871290 1176 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:36.871974 1210 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:36.960492 1026 raft_consensus.cc:1261] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:36.961795 1241 consensus_queue.cc:1035] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:36.970996 1101 raft_consensus.cc:1261] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:36.972985 1176 consensus_queue.cc:1035] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:36:37.165036 1101 raft_consensus.cc:1261] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:37.166414 1199 consensus_queue.cc:1035] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:37.172821 953 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:37.175854 1151 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:37.176105 953 raft_consensus.cc:1261] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:37.178313 1199 consensus_queue.cc:1035] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:36:37.190788 1101 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:37.194897 1151 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:36:39.406143 818 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260927 19:36:39.406949 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:39.431279 953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
mode: GRACEFUL
new_leader_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:40072
I20260927 19:36:39.431912 953 raft_consensus.cc:595] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Received request to transfer leadership to TS 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:39.444603 1026 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:47840
I20260927 19:36:39.445261 1026 raft_consensus.cc:595] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Received request to transfer leadership to TS 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:39.446817 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260927 19:36:39.447398 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:39.629027 1254 raft_consensus.cc:979] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: : Instructing follower 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:39.629460 1254 raft_consensus.cc:1067] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Signalling peer 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:39.631002 1102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:40992
I20260927 19:36:39.631453 1102 raft_consensus.cc:482] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:39.631696 1102 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:39.636636 1102 raft_consensus.cc:504] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:39.638794 1102 leader_election.cc:288] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 2 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395)
I20260927 19:36:39.639622 1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:39.639882 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:39.640244 1026 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:39.640395 953 raft_consensus.cc:3032] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:39.640733 953 raft_consensus.cc:729] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:39.641536 953 consensus_queue.cc:260] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:39.642750 953 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:39.647249 1026 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 2.
I20260927 19:36:39.648943 1067 leader_election.cc:302] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:39.650410 1153 raft_consensus.cc:2781] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:39.650406 953 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 2.
I20260927 19:36:39.652033 1153 raft_consensus.cc:686] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Becoming Leader. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:39.652911 1153 consensus_queue.cc:237] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:39.662884 847 catalog_manager.cc:5495] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 reported cstate change: term changed from 1 to 2, leader changed from 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129) to 44c79fa05e27477da4ec1b913410f533 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "44c79fa05e27477da4ec1b913410f533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:39.884416 1261 raft_consensus.cc:979] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:39.884986 1199 raft_consensus.cc:1067] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Signalling peer 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:39.886785 1102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:40988
I20260927 19:36:39.887321 1102 raft_consensus.cc:482] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:39.887677 1102 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:39.892508 1102 raft_consensus.cc:504] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:39.894490 1102 leader_election.cc:288] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 3 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:39.895450 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:39.895854 1026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:39.896100 953 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:39.896461 1026 raft_consensus.cc:3032] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:36:39.896730 1026 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:39.897431 1026 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:39.898311 1026 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:39.901201 953 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 3.
I20260927 19:36:39.902853 1066 leader_election.cc:302] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:39.903661 1153 raft_consensus.cc:2781] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:36:39.903681 1026 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 3.
I20260927 19:36:39.904223 1153 raft_consensus.cc:686] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 LEADER]: Becoming Leader. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:39.905020 1153 consensus_queue.cc:237] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:39.912017 847 catalog_manager.cc:5495] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 reported cstate change: term changed from 2 to 3, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 44c79fa05e27477da4ec1b913410f533 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "44c79fa05e27477da4ec1b913410f533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260927 19:36:40.242466 1026 raft_consensus.cc:1261] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:40.243918 1153 consensus_queue.cc:1035] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:40.260038 953 raft_consensus.cc:1261] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:40.261595 1153 consensus_queue.cc:1035] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:40.331995 1026 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:40.333259 1255 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:40.345427 953 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:40.346872 1266 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:36:40.448309 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:40.452927 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:40.453293 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:41.454129 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:41.458966 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:41.459323 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:42.460278 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:42.467595 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:42.468130 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:43.468987 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:43.474725 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:43.475134 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:44.475989 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:44.481137 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:44.481626 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:45.482427 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:45.487015 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:45.487457 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:46.488286 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:46.493400 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:46.493788 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:47.494668 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:47.499487 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:47.499899 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:48.500818 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:48.505838 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:48.506253 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:49.534924 818 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260927 19:36:49.539501 1101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:41006
I20260927 19:36:49.540135 1101 raft_consensus.cc:595] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:49.544754 953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:40922
I20260927 19:36:49.545300 953 raft_consensus.cc:595] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:49.551010 953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:40922
I20260927 19:36:49.551553 953 raft_consensus.cc:595] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:49.556192 953 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:40922
I20260927 19:36:49.556757 953 raft_consensus.cc:595] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:49.564874 1101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
mode: GRACEFUL
new_leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:41006
I20260927 19:36:49.565503 1101 raft_consensus.cc:595] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 LEADER]: Received request to transfer leadership to TS 2d0b4f3ae70e4858a14b92f93ecda542
I20260927 19:36:49.656209 1281 raft_consensus.cc:979] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.656649 1281 raft_consensus.cc:1067] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.658277 1027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56328
I20260927 19:36:49.658855 1027 raft_consensus.cc:482] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:49.659245 1027 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.664191 1027 raft_consensus.cc:504] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:49.665865 1027 leader_election.cc:288] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 3 election: Requested vote from peers 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069), 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395)
I20260927 19:36:49.666999 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:49.666970 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1bdd6d3afaf4de6ba4f3a0a9288b81c" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:49.667676 953 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.667754 1101 raft_consensus.cc:3032] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:36:49.668147 1101 raft_consensus.cc:729] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:49.668751 1101 consensus_queue.cc:260] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:49.669795 1101 raft_consensus.cc:3037] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.672307 953 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 3.
I20260927 19:36:49.673689 992 leader_election.cc:302] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:49.674463 1272 raft_consensus.cc:2781] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:36:49.675030 1101 raft_consensus.cc:2454] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 3.
I20260927 19:36:49.675379 1272 raft_consensus.cc:686] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:49.676437 1272 consensus_queue.cc:237] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } }
I20260927 19:36:49.683430 847 catalog_manager.cc:5495] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 2 to 3, leader changed from 44c79fa05e27477da4ec1b913410f533 (127.0.204.131) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:49.698715 1270 raft_consensus.cc:979] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.699216 1280 raft_consensus.cc:1067] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.700754 1027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56320
I20260927 19:36:49.701272 1027 raft_consensus.cc:482] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:49.701650 1027 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.705967 1027 raft_consensus.cc:504] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.707574 1027 leader_election.cc:288] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 3 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:49.708595 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:49.708407 953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:49.709201 1101 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.709257 953 raft_consensus.cc:3032] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:36:49.709960 953 raft_consensus.cc:729] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:49.711241 1270 raft_consensus.cc:979] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.711669 1270 raft_consensus.cc:1067] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.710772 953 consensus_queue.cc:260] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.713121 1027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56320
I20260927 19:36:49.713711 953 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.713667 1027 raft_consensus.cc:482] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:49.714218 1027 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:49.715322 1101 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 3.
I20260927 19:36:49.716403 992 leader_election.cc:302] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:49.717171 1272 raft_consensus.cc:2781] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:36:49.717687 1272 raft_consensus.cc:686] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:49.718755 1272 consensus_queue.cc:237] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.720535 953 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 3.
I20260927 19:36:49.720845 1027 raft_consensus.cc:504] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.723058 1027 leader_election.cc:288] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:49.724108 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:49.724553 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:49.724794 1101 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:49.725118 952 raft_consensus.cc:3032] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:49.725495 952 raft_consensus.cc:729] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:49.726294 952 consensus_queue.cc:260] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.727634 952 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:49.730202 847 catalog_manager.cc:5495] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 2 to 3, leader changed from 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:49.732295 952 raft_consensus.cc:2454] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:49.732326 1101 raft_consensus.cc:2454] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 2.
I20260927 19:36:49.733301 992 leader_election.cc:302] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:49.733989 1272 raft_consensus.cc:2781] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:49.734433 1272 raft_consensus.cc:686] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:49.735255 1272 consensus_queue.cc:237] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.743024 847 catalog_manager.cc:5495] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 1 to 2, leader changed from 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:49.804908 1280 raft_consensus.cc:979] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.805505 1270 raft_consensus.cc:1067] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.807174 1027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56320
I20260927 19:36:49.807705 1027 raft_consensus.cc:482] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:49.808041 1027 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.813284 1027 raft_consensus.cc:504] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.815049 1027 leader_election.cc:288] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 3 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:49.816028 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:49.816228 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685c5d66a10549479f13281fcd7fbbc4" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:49.816797 952 raft_consensus.cc:3032] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:36:49.816962 1101 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.817267 952 raft_consensus.cc:729] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:49.818212 952 consensus_queue.cc:260] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.819223 952 raft_consensus.cc:3037] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:49.821570 1101 raft_consensus.cc:2454] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 3.
I20260927 19:36:49.822506 992 leader_election.cc:302] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0b4f3ae70e4858a14b92f93ecda542, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:49.823168 1272 raft_consensus.cc:2781] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:36:49.823676 1272 raft_consensus.cc:686] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:49.825044 952 raft_consensus.cc:2454] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 3.
I20260927 19:36:49.824721 1272 consensus_queue.cc:237] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.831701 847 catalog_manager.cc:5495] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 2 to 3, leader changed from 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:49.910744 1281 raft_consensus.cc:979] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: : Instructing follower 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.911263 1281 raft_consensus.cc:1067] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 LEADER]: Signalling peer 2d0b4f3ae70e4858a14b92f93ecda542 to start an election
I20260927 19:36:49.912909 1027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
from {username='slave'} at 127.0.0.1:56328
I20260927 19:36:49.913496 1027 raft_consensus.cc:482] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:49.913904 1027 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:36:49.918721 1027 raft_consensus.cc:504] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.921205 1027 leader_election.cc:288] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 4 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:49.922156 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:49.922340 1101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:49.922792 952 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:36:49.922808 1101 raft_consensus.cc:3032] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:36:49.923334 1101 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:49.924310 1101 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.925645 1101 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:36:49.929895 952 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 4.
I20260927 19:36:49.931036 992 leader_election.cc:302] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 2d0b4f3ae70e4858a14b92f93ecda542; no voters:
I20260927 19:36:49.931825 1272 raft_consensus.cc:2781] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:36:49.932229 1101 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d0b4f3ae70e4858a14b92f93ecda542 in term 4.
I20260927 19:36:49.932415 1272 raft_consensus.cc:686] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 LEADER]: Becoming Leader. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:49.933507 1272 consensus_queue.cc:237] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:49.940572 847 catalog_manager.cc:5495] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 reported cstate change: term changed from 3 to 4, leader changed from 44c79fa05e27477da4ec1b913410f533 (127.0.204.131) to 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130). New cstate: current_term: 4 leader_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:50.146510 1101 raft_consensus.cc:1261] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:50.147812 1272 consensus_queue.cc:1035] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:50.158084 952 raft_consensus.cc:1261] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:50.159685 1294 consensus_queue.cc:1035] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:36:50.207437 952 raft_consensus.cc:1261] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:50.208693 1294 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:50.218438 1101 raft_consensus.cc:1261] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:50.220453 1294 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:50.226533 1102 raft_consensus.cc:1261] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:50.227670 1294 consensus_queue.cc:1035] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:50.236114 952 raft_consensus.cc:1261] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:50.238598 1301 consensus_queue.cc:1035] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:36:50.250032 1102 raft_consensus.cc:1261] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:50.251600 1301 consensus_queue.cc:1035] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:50.262154 952 raft_consensus.cc:1261] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:50.263358 1272 consensus_queue.cc:1035] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:50.508378 952 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:36:50.509764 1272 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:36:50.518868 1102 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Refusing update from remote peer 2d0b4f3ae70e4858a14b92f93ecda542: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:36:50.520383 1272 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:36:52.605710 818 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260927 19:36:52.606492 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:52.614110 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:47840
I20260927 19:36:52.614945 1027 raft_consensus.cc:595] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Received request to transfer leadership to TS 1825fc1c3aa64ba5a098b1d7c734e59c
I20260927 19:36:52.617833 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:47840
I20260927 19:36:52.618273 1027 raft_consensus.cc:595] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Received request to transfer leadership to TS 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:52.620437 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:47840
I20260927 19:36:52.620889 1027 raft_consensus.cc:595] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Received request to transfer leadership to TS 1825fc1c3aa64ba5a098b1d7c734e59c
I20260927 19:36:52.622948 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:47840
I20260927 19:36:52.623405 1027 raft_consensus.cc:595] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Received request to transfer leadership to TS 1825fc1c3aa64ba5a098b1d7c734e59c
I20260927 19:36:52.625609 1027 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
mode: GRACEFUL
new_leader_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:47840
I20260927 19:36:52.626019 1027 raft_consensus.cc:595] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 LEADER]: Received request to transfer leadership to TS 44c79fa05e27477da4ec1b913410f533
I20260927 19:36:52.626967 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:36:52.627436 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:52.651876 1312 raft_consensus.cc:979] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:52.652504 1282 raft_consensus.cc:1067] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Signalling peer 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:52.654022 952 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:40902
I20260927 19:36:52.654485 952 raft_consensus.cc:482] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:52.654790 952 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:36:52.660899 952 raft_consensus.cc:504] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.663219 952 leader_election.cc:288] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 4 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:52.664503 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:52.664248 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab1070fbc6e48bca66b39aa1a213162" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:52.665072 1102 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:36:52.665167 1027 raft_consensus.cc:3032] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:36:52.665609 1027 raft_consensus.cc:729] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:52.666347 1027 consensus_queue.cc:260] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.667455 1027 raft_consensus.cc:3037] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:36:52.669682 1102 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 4.
I20260927 19:36:52.670811 917 leader_election.cc:302] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:52.672411 1313 raft_consensus.cc:2781] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:36:52.673204 1027 raft_consensus.cc:2454] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 4.
I20260927 19:36:52.673435 1313 raft_consensus.cc:686] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:52.674270 1313 consensus_queue.cc:237] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.681497 849 catalog_manager.cc:5495] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 3 to 4, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 4 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:52.807183 1312 raft_consensus.cc:979] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:52.807592 1282 raft_consensus.cc:1067] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Signalling peer 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:52.808946 1102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:40988
I20260927 19:36:52.809329 1102 raft_consensus.cc:482] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:52.809623 1102 raft_consensus.cc:3037] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:52.813519 1102 raft_consensus.cc:504] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.815142 1102 leader_election.cc:288] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 2 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:52.816054 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:52.816054 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86003e86d9224dd6a67bef7f95cc4881" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:52.816766 1027 raft_consensus.cc:3032] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:36:52.816841 952 raft_consensus.cc:3037] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:52.817150 1027 raft_consensus.cc:729] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:52.817878 1027 consensus_queue.cc:260] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.819065 1027 raft_consensus.cc:3037] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:36:52.823397 952 raft_consensus.cc:2454] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 2.
I20260927 19:36:52.824699 1066 leader_election.cc:302] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:52.825295 1027 raft_consensus.cc:2454] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 2.
I20260927 19:36:52.825665 1309 raft_consensus.cc:2781] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:36:52.826741 1309 raft_consensus.cc:686] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Becoming Leader. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:52.827425 1309 consensus_queue.cc:237] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.834482 849 catalog_manager.cc:5495] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 reported cstate change: term changed from 1 to 2, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 44c79fa05e27477da4ec1b913410f533 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "44c79fa05e27477da4ec1b913410f533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260927 19:36:52.873036 1312 raft_consensus.cc:979] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:52.873570 1282 raft_consensus.cc:1067] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Signalling peer 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:52.874964 952 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:40902
I20260927 19:36:52.875519 952 raft_consensus.cc:482] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:52.875861 952 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:52.880632 952 raft_consensus.cc:504] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.882083 952 leader_election.cc:288] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 3 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:52.883002 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:52.883042 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994ef439b09b46d3abcf439e645045e4" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:52.883672 1027 raft_consensus.cc:3032] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:36:52.883697 1102 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:52.884122 1027 raft_consensus.cc:729] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:52.884778 1027 consensus_queue.cc:260] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.885879 1027 raft_consensus.cc:3037] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:52.889859 1102 raft_consensus.cc:2454] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 3.
I20260927 19:36:52.890883 917 leader_election.cc:302] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:52.891642 1313 raft_consensus.cc:2781] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:36:52.891784 1027 raft_consensus.cc:2454] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 3.
I20260927 19:36:52.892215 1313 raft_consensus.cc:686] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:52.893601 1313 consensus_queue.cc:237] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.900346 849 catalog_manager.cc:5495] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 2 to 3, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 3 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:52.927647 1312 raft_consensus.cc:979] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:52.928149 1282 raft_consensus.cc:1067] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Signalling peer 1825fc1c3aa64ba5a098b1d7c734e59c to start an election
I20260927 19:36:52.929477 952 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081"
dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
from {username='slave'} at 127.0.0.1:40902
I20260927 19:36:52.929975 952 raft_consensus.cc:482] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:52.930210 952 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:52.934088 952 raft_consensus.cc:504] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.935539 952 leader_election.cc:288] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 3 election: Requested vote from peers 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583), 44c79fa05e27477da4ec1b913410f533 (127.0.204.131:36069)
I20260927 19:36:52.936321 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:52.936462 1102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8501155ee43d4303b49dd91872fa6081" candidate_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44c79fa05e27477da4ec1b913410f533"
I20260927 19:36:52.936852 1027 raft_consensus.cc:3032] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:36:52.936932 1102 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:52.937127 1027 raft_consensus.cc:729] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:52.937791 1027 consensus_queue.cc:260] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.938575 1027 raft_consensus.cc:3037] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:36:52.941000 1102 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 3.
I20260927 19:36:52.941978 917 leader_election.cc:302] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:52.942638 1313 raft_consensus.cc:2781] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:36:52.942883 1027 raft_consensus.cc:2454] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1825fc1c3aa64ba5a098b1d7c734e59c in term 3.
I20260927 19:36:52.943102 1313 raft_consensus.cc:686] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 LEADER]: Becoming Leader. State: Replica: 1825fc1c3aa64ba5a098b1d7c734e59c, State: Running, Role: LEADER
I20260927 19:36:52.943881 1313 consensus_queue.cc:237] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:52.950546 849 catalog_manager.cc:5495] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c reported cstate change: term changed from 2 to 3, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129). New cstate: current_term: 3 leader_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: UNKNOWN } } }
I20260927 19:36:53.017726 1312 raft_consensus.cc:979] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: : Instructing follower 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:53.018252 1282 raft_consensus.cc:1067] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 LEADER]: Signalling peer 44c79fa05e27477da4ec1b913410f533 to start an election
I20260927 19:36:53.019578 1102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968"
dest_uuid: "44c79fa05e27477da4ec1b913410f533"
from {username='slave'} at 127.0.0.1:40988
I20260927 19:36:53.020053 1102 raft_consensus.cc:482] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:36:53.020345 1102 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:36:53.024389 1102 raft_consensus.cc:504] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:53.025743 1102 leader_election.cc:288] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 5 election: Requested vote from peers 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395), 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130:40583)
I20260927 19:36:53.026712 1027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2d0b4f3ae70e4858a14b92f93ecda542"
I20260927 19:36:53.026648 952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f2925a8b52418c87249ef2c5743968" candidate_uuid: "44c79fa05e27477da4ec1b913410f533" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c"
I20260927 19:36:53.027285 1027 raft_consensus.cc:3032] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:36:53.027297 952 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:36:53.027701 1027 raft_consensus.cc:729] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2d0b4f3ae70e4858a14b92f93ecda542, State: Running, Role: LEADER
I20260927 19:36:53.028386 1027 consensus_queue.cc:260] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:53.029510 1027 raft_consensus.cc:3037] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:36:53.033608 952 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 5.
I20260927 19:36:53.034837 1066 leader_election.cc:302] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1825fc1c3aa64ba5a098b1d7c734e59c, 44c79fa05e27477da4ec1b913410f533; no voters:
I20260927 19:36:53.035120 1027 raft_consensus.cc:2454] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c79fa05e27477da4ec1b913410f533 in term 5.
I20260927 19:36:53.035535 1309 raft_consensus.cc:2781] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:36:53.036033 1309 raft_consensus.cc:686] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 5 LEADER]: Becoming Leader. State: Replica: 44c79fa05e27477da4ec1b913410f533, State: Running, Role: LEADER
I20260927 19:36:53.036804 1309 consensus_queue.cc:237] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } }
I20260927 19:36:53.044169 849 catalog_manager.cc:5495] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 reported cstate change: term changed from 4 to 5, leader changed from 2d0b4f3ae70e4858a14b92f93ecda542 (127.0.204.130) to 44c79fa05e27477da4ec1b913410f533 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "44c79fa05e27477da4ec1b913410f533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260927 19:36:53.089993 1102 raft_consensus.cc:1261] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:36:53.091461 1313 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:36:53.103026 1027 raft_consensus.cc:1261] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:36:53.104542 1313 consensus_queue.cc:1035] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:36:53.293958 1027 raft_consensus.cc:1261] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:53.295150 1313 consensus_queue.cc:1035] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:53.303840 1102 raft_consensus.cc:1261] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:53.305461 1314 consensus_queue.cc:1035] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:36:53.367192 1027 raft_consensus.cc:1261] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:53.368353 1309 consensus_queue.cc:1035] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:36:53.377096 952 raft_consensus.cc:1261] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:36:53.379381 1334 consensus_queue.cc:1035] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:36:53.407567 1027 raft_consensus.cc:1261] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:53.408692 1314 consensus_queue.cc:1035] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:53.418082 1102 raft_consensus.cc:1261] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Refusing update from remote peer 1825fc1c3aa64ba5a098b1d7c734e59c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:36:53.419376 1313 consensus_queue.cc:1035] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44c79fa05e27477da4ec1b913410f533" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:36:53.443616 952 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 5 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:36:53.445024 1334 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1825fc1c3aa64ba5a098b1d7c734e59c" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:36:53.454196 1027 raft_consensus.cc:1261] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 5 FOLLOWER]: Refusing update from remote peer 44c79fa05e27477da4ec1b913410f533: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:36:53.455689 1309 consensus_queue.cc:1035] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0b4f3ae70e4858a14b92f93ecda542" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:36:53.628299 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:53.632820 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:53.633252 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:54.634148 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:54.638823 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:54.639279 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:55.640085 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:55.644652 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:55.645089 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:56.645895 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:56.651031 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:56.651505 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:57.652427 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:57.657630 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:57.658262 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:58.659194 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:58.663822 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:58.664247 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:36:59.664991 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:36:59.669891 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:36:59.670323 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:00.671096 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:00.676518 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:00.676961 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:01.677903 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:01.682853 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:01.683274 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:02.715070 818 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260927 19:37:02.715837 818 tablet_server.cc:178] TabletServer@127.0.204.129:0 shutting down...
I20260927 19:37:02.749919 818 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 19:37:02.750897 818 tablet_replica.cc:330] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.751662 818 raft_consensus.cc:2229] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.752355 818 raft_consensus.cc:2258] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.754720 818 tablet_replica.cc:330] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.755263 818 raft_consensus.cc:2229] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.755681 818 raft_consensus.cc:2258] T 7be07c6c716147659d99a15bc41453d0 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.757293 818 tablet_replica.cc:330] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.757843 818 raft_consensus.cc:2229] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.758260 818 raft_consensus.cc:2258] T 10dbe8ad420c4a2b97851e08f06d7d5d P 1825fc1c3aa64ba5a098b1d7c734e59c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.759912 818 tablet_replica.cc:330] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.760423 818 raft_consensus.cc:2229] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 LEADER]: Raft consensus shutting down.
I20260927 19:37:02.761408 818 raft_consensus.cc:2258] T 9ab1070fbc6e48bca66b39aa1a213162 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.763188 818 tablet_replica.cc:330] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.764074 818 raft_consensus.cc:2229] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.764605 818 raft_consensus.cc:2258] T c6f2925a8b52418c87249ef2c5743968 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.766419 818 tablet_replica.cc:330] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.766925 818 raft_consensus.cc:2229] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 LEADER]: Raft consensus shutting down.
I20260927 19:37:02.767727 818 raft_consensus.cc:2258] T 8501155ee43d4303b49dd91872fa6081 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.769449 818 tablet_replica.cc:330] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.769908 818 raft_consensus.cc:2229] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.770344 818 raft_consensus.cc:2258] T 86003e86d9224dd6a67bef7f95cc4881 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.771976 818 tablet_replica.cc:330] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.772437 818 raft_consensus.cc:2229] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.772852 818 raft_consensus.cc:2258] T 685c5d66a10549479f13281fcd7fbbc4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.774521 818 tablet_replica.cc:330] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c: stopping tablet replica
I20260927 19:37:02.774969 818 raft_consensus.cc:2229] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 LEADER]: Raft consensus shutting down.
I20260927 19:37:02.775758 818 raft_consensus.cc:2258] T 994ef439b09b46d3abcf439e645045e4 P 1825fc1c3aa64ba5a098b1d7c734e59c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.872916 818 tablet_server.cc:194] TabletServer@127.0.204.129:0 shutdown complete.
W20260927 19:37:02.890612 1066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.129:40395: connect: Connection refused (error 111)
W20260927 19:37:02.894860 1066 consensus_peers.cc:487] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 -> Peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Couldn't send request to peer 1825fc1c3aa64ba5a098b1d7c734e59c. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.129:40395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:02.896355 818 tablet_server.cc:178] TabletServer@127.0.204.130:0 shutting down...
W20260927 19:37:02.913324 992 consensus_peers.cc:487] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 -> Peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Couldn't send request to peer 1825fc1c3aa64ba5a098b1d7c734e59c. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.129:40395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260927 19:37:02.929505 992 consensus_peers.cc:487] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 -> Peer 1825fc1c3aa64ba5a098b1d7c734e59c (127.0.204.129:40395): Couldn't send request to peer 1825fc1c3aa64ba5a098b1d7c734e59c. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.129:40395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:02.930064 818 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 19:37:02.930828 818 tablet_replica.cc:330] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.931648 818 raft_consensus.cc:2229] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:02.932482 818 raft_consensus.cc:2258] T 7be07c6c716147659d99a15bc41453d0 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.934329 818 tablet_replica.cc:330] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.934899 818 raft_consensus.cc:2229] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.935453 818 raft_consensus.cc:2258] T 994ef439b09b46d3abcf439e645045e4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.937644 818 tablet_replica.cc:330] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.938196 818 raft_consensus.cc:2229] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.938606 818 raft_consensus.cc:2258] T c6f2925a8b52418c87249ef2c5743968 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.940194 818 tablet_replica.cc:330] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.940688 818 raft_consensus.cc:2229] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.941110 818 raft_consensus.cc:2258] T 9ab1070fbc6e48bca66b39aa1a213162 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.942716 818 tablet_replica.cc:330] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.943218 818 raft_consensus.cc:2229] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Raft consensus shutting down.
I20260927 19:37:02.943990 818 raft_consensus.cc:2258] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.945693 818 tablet_replica.cc:330] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.946573 818 raft_consensus.cc:2229] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.947077 818 raft_consensus.cc:2258] T 8501155ee43d4303b49dd91872fa6081 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.948839 818 tablet_replica.cc:330] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.949462 818 raft_consensus.cc:2229] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.949939 818 raft_consensus.cc:2258] T 86003e86d9224dd6a67bef7f95cc4881 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.951627 818 tablet_replica.cc:330] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.952127 818 raft_consensus.cc:2229] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 LEADER]: Raft consensus shutting down.
I20260927 19:37:02.953148 818 raft_consensus.cc:2258] T 685c5d66a10549479f13281fcd7fbbc4 P 2d0b4f3ae70e4858a14b92f93ecda542 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.955068 818 tablet_replica.cc:330] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542: stopping tablet replica
I20260927 19:37:02.955570 818 raft_consensus.cc:2229] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:02.955999 818 raft_consensus.cc:2258] T 10dbe8ad420c4a2b97851e08f06d7d5d P 2d0b4f3ae70e4858a14b92f93ecda542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:02.986737 818 tablet_server.cc:194] TabletServer@127.0.204.130:0 shutdown complete.
I20260927 19:37:03.004777 818 tablet_server.cc:178] TabletServer@127.0.204.131:0 shutting down...
I20260927 19:37:03.029680 818 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 19:37:03.030448 818 tablet_replica.cc:330] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.031152 818 raft_consensus.cc:2229] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:03.031667 818 raft_consensus.cc:2258] T a1bdd6d3afaf4de6ba4f3a0a9288b81c P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.033437 818 tablet_replica.cc:330] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.033941 818 raft_consensus.cc:2229] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:03.034369 818 raft_consensus.cc:2258] T 7be07c6c716147659d99a15bc41453d0 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.035965 818 tablet_replica.cc:330] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.036460 818 raft_consensus.cc:2229] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:03.036926 818 raft_consensus.cc:2258] T 9ab1070fbc6e48bca66b39aa1a213162 P 44c79fa05e27477da4ec1b913410f533 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.038576 818 tablet_replica.cc:330] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.039064 818 raft_consensus.cc:2229] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 5 LEADER]: Raft consensus shutting down.
I20260927 19:37:03.039840 818 raft_consensus.cc:2258] T c6f2925a8b52418c87249ef2c5743968 P 44c79fa05e27477da4ec1b913410f533 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.041571 818 tablet_replica.cc:330] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.042070 818 raft_consensus.cc:2229] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 1 LEADER]: Raft consensus shutting down.
I20260927 19:37:03.042811 818 raft_consensus.cc:2258] T 10dbe8ad420c4a2b97851e08f06d7d5d P 44c79fa05e27477da4ec1b913410f533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.044382 818 tablet_replica.cc:330] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.044872 818 raft_consensus.cc:2229] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:03.045400 818 raft_consensus.cc:2258] T 994ef439b09b46d3abcf439e645045e4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.047093 818 tablet_replica.cc:330] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.047562 818 raft_consensus.cc:2229] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:03.048333 818 raft_consensus.cc:2258] T 86003e86d9224dd6a67bef7f95cc4881 P 44c79fa05e27477da4ec1b913410f533 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.050024 818 tablet_replica.cc:330] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.050448 818 raft_consensus.cc:2229] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:03.050901 818 raft_consensus.cc:2258] T 685c5d66a10549479f13281fcd7fbbc4 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.052507 818 tablet_replica.cc:330] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533: stopping tablet replica
I20260927 19:37:03.053026 818 raft_consensus.cc:2229] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:03.053519 818 raft_consensus.cc:2258] T 8501155ee43d4303b49dd91872fa6081 P 44c79fa05e27477da4ec1b913410f533 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.081645 818 tablet_server.cc:194] TabletServer@127.0.204.131:0 shutdown complete.
I20260927 19:37:03.100126 818 master.cc:537] Master@127.0.204.190:37763 shutting down...
I20260927 19:37:03.217938 818 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 1 LEADER]: Raft consensus shutting down.
I20260927 19:37:03.218566 818 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:03.218976 818 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8ffcb64138d74550a6ffff66e887f411: stopping tablet replica
I20260927 19:37:03.238126 818 master.cc:552] Master@127.0.204.190:37763 shutdown complete.
1825fc1c3aa64ba5a098b1d7c734e59c 4
2d0b4f3ae70e4858a14b92f93ecda542 4
44c79fa05e27477da4ec1b913410f533 1
1825fc1c3aa64ba5a098b1d7c734e59c 3
2d0b4f3ae70e4858a14b92f93ecda542 3
44c79fa05e27477da4ec1b913410f533 3
1825fc1c3aa64ba5a098b1d7c734e59c 0
2d0b4f3ae70e4858a14b92f93ecda542 8
44c79fa05e27477da4ec1b913410f533 1
1825fc1c3aa64ba5a098b1d7c734e59c 3
2d0b4f3ae70e4858a14b92f93ecda542 3
44c79fa05e27477da4ec1b913410f533 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31488 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260927 19:37:03.283738 818 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.204.190:36435
I20260927 19:37:03.284787 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:03.290779 1353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.291342 1354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.292065 1356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:03.292649 818 server_base.cc:1034] running on GCE node
I20260927 19:37:03.293883 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:03.294064 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:03.294236 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537823294222 us; error 0 us; skew 500 ppm
I20260927 19:37:03.294838 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:03.302146 818 webserver.cc:460] Webserver started at http://127.0.204.190:45999/ using document root <none> and password file <none>
I20260927 19:37:03.302726 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:03.302919 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:03.303252 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:03.304476 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/master-0-root/instance:
uuid: "7fd0855939b84ca1a4a229994b18a745"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.309523 818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 19:37:03.313050 1361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.314025 818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 19:37:03.314370 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/master-0-root
uuid: "7fd0855939b84ca1a4a229994b18a745"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.314661 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:03.331707 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:03.332950 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:03.372414 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.190:36435
I20260927 19:37:03.372532 1412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.190:36435 every 8 connection(s)
I20260927 19:37:03.376668 1413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.387082 1413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745: Bootstrap starting.
I20260927 19:37:03.391338 1413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:03.395558 1413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745: No bootstrap required, opened a new log
I20260927 19:37:03.397791 1413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd0855939b84ca1a4a229994b18a745" member_type: VOTER }
I20260927 19:37:03.398240 1413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:03.398411 1413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd0855939b84ca1a4a229994b18a745, State: Initialized, Role: FOLLOWER
I20260927 19:37:03.399024 1413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd0855939b84ca1a4a229994b18a745" member_type: VOTER }
I20260927 19:37:03.399442 1413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 19:37:03.399623 1413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 19:37:03.399842 1413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:03.403621 1413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd0855939b84ca1a4a229994b18a745" member_type: VOTER }
I20260927 19:37:03.404083 1413 leader_election.cc:302] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd0855939b84ca1a4a229994b18a745; no voters:
I20260927 19:37:03.405118 1413 leader_election.cc:288] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 19:37:03.405683 1416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:03.407116 1416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 1 LEADER]: Becoming Leader. State: Replica: 7fd0855939b84ca1a4a229994b18a745, State: Running, Role: LEADER
I20260927 19:37:03.407814 1416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd0855939b84ca1a4a229994b18a745" member_type: VOTER }
I20260927 19:37:03.408027 1413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 19:37:03.414469 1417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd0855939b84ca1a4a229994b18a745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd0855939b84ca1a4a229994b18a745" member_type: VOTER } }
I20260927 19:37:03.414633 1418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd0855939b84ca1a4a229994b18a745. Latest consensus state: current_term: 1 leader_uuid: "7fd0855939b84ca1a4a229994b18a745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd0855939b84ca1a4a229994b18a745" member_type: VOTER } }
I20260927 19:37:03.415241 1417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [sys.catalog]: This master's current role is: LEADER
I20260927 19:37:03.415369 1418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [sys.catalog]: This master's current role is: LEADER
I20260927 19:37:03.417584 1424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 19:37:03.424606 1424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 19:37:03.425763 818 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260927 19:37:03.434599 1424 catalog_manager.cc:1347] Generated new cluster ID: b1df90ad88fe477cac4027fc0e04acc5
I20260927 19:37:03.434957 1424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 19:37:03.462831 1424 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 19:37:03.464551 1424 catalog_manager.cc:1504] Loading token signing keys...
I20260927 19:37:03.478896 1424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745: Generated new TSK 0
I20260927 19:37:03.479697 1424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 19:37:03.492468 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:03.499035 1434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.499650 1435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.502072 1437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:03.503074 818 server_base.cc:1034] running on GCE node
I20260927 19:37:03.504043 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:03.504230 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:03.504431 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537823504400 us; error 0 us; skew 500 ppm
I20260927 19:37:03.504971 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:03.507699 818 webserver.cc:460] Webserver started at http://127.0.204.129:33181/ using document root <none> and password file <none>
I20260927 19:37:03.508206 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:03.508370 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:03.508577 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:03.509838 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-0-root/instance:
uuid: "2703f921034946859b3f26666c525028"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.514566 818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 19:37:03.518527 1442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.519512 818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 19:37:03.519855 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-0-root
uuid: "2703f921034946859b3f26666c525028"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.520157 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:03.533995 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:03.535163 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:03.536629 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:03.538952 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:03.539177 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.539417 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:03.539600 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.579799 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.129:45337
I20260927 19:37:03.579886 1504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.129:45337 every 8 connection(s)
I20260927 19:37:03.584482 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:03.592166 1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.592635 1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:03.596535 818 server_base.cc:1034] running on GCE node
W20260927 19:37:03.598554 1513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:03.599573 1505 heartbeater.cc:347] Connected to a master server at 127.0.204.190:36435
I20260927 19:37:03.599606 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:03.600044 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:03.600085 1505 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:03.600248 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537823600231 us; error 0 us; skew 500 ppm
I20260927 19:37:03.600878 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:03.601007 1505 heartbeater.cc:511] Master 127.0.204.190:36435 requested a full tablet report, sending...
I20260927 19:37:03.603142 1378 ts_manager.cc:194] Registered new tserver with Master: 2703f921034946859b3f26666c525028 (127.0.204.129:45337)
I20260927 19:37:03.603808 818 webserver.cc:460] Webserver started at http://127.0.204.130:43211/ using document root <none> and password file <none>
I20260927 19:37:03.604322 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:03.604511 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:03.604772 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:03.605257 1378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42400
I20260927 19:37:03.606007 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-1-root/instance:
uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.610744 818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 19:37:03.614213 1517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.615052 818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 19:37:03.615371 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-1-root
uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.615707 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:03.635671 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:03.636857 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:03.638322 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:03.640491 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:03.640720 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.640964 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:03.641125 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.680111 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.130:46457
I20260927 19:37:03.680298 1579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.130:46457 every 8 connection(s)
I20260927 19:37:03.684772 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:03.692914 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.694638 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:03.695629 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:03.696230 818 server_base.cc:1034] running on GCE node
I20260927 19:37:03.697208 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:03.697499 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:03.697695 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537823697678 us; error 0 us; skew 500 ppm
I20260927 19:37:03.698204 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:03.700423 1580 heartbeater.cc:347] Connected to a master server at 127.0.204.190:36435
I20260927 19:37:03.700882 1580 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:03.700925 818 webserver.cc:460] Webserver started at http://127.0.204.131:35635/ using document root <none> and password file <none>
I20260927 19:37:03.701610 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:03.701701 1580 heartbeater.cc:511] Master 127.0.204.190:36435 requested a full tablet report, sending...
I20260927 19:37:03.701840 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:03.702196 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:03.703621 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-2-root/instance:
uuid: "2a8943ab8566448487437f6561e08900"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.703876 1378 ts_manager.cc:194] Registered new tserver with Master: 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:03.705600 1378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42402
I20260927 19:37:03.708742 818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 19:37:03.712476 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.713192 818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 19:37:03.713515 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-2-root
uuid: "2a8943ab8566448487437f6561e08900"
format_stamp: "Formatted at 2026-09-27 19:37:03 on dist-test-slave-zwk0"
I20260927 19:37:03.713842 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790537791628671-818-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:03.754992 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:03.756508 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:03.758118 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:03.760411 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:03.760640 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.760892 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:03.761058 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:03.802700 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.131:36223
I20260927 19:37:03.802875 1653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.131:36223 every 8 connection(s)
I20260927 19:37:03.819867 1654 heartbeater.cc:347] Connected to a master server at 127.0.204.190:36435
I20260927 19:37:03.820290 1654 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:03.821101 1654 heartbeater.cc:511] Master 127.0.204.190:36435 requested a full tablet report, sending...
I20260927 19:37:03.823055 1378 ts_manager.cc:194] Registered new tserver with Master: 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:03.823292 818 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017104827s
I20260927 19:37:03.824755 1378 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42418
I20260927 19:37:03.826030 818 test_util.cc:268] Using random seed: 317861972
I20260927 19:37:03.852818 1378 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42424:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 19:37:03.855732 1378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 19:37:03.934013 1619 tablet_service.cc:1467] Processing CreateTablet for tablet a89edbac45f14ad399aafe8ea1932989 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260927 19:37:03.934017 1618 tablet_service.cc:1467] Processing CreateTablet for tablet cc5273de00ed4b70b5ee53cabc03c689 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260927 19:37:03.935634 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89edbac45f14ad399aafe8ea1932989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.936491 1618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5273de00ed4b70b5ee53cabc03c689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.939131 1545 tablet_service.cc:1467] Processing CreateTablet for tablet a89edbac45f14ad399aafe8ea1932989 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260927 19:37:03.940490 1545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89edbac45f14ad399aafe8ea1932989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.941044 1617 tablet_service.cc:1467] Processing CreateTablet for tablet 960861c418c145508e7dc07cc2e571c3 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260927 19:37:03.941555 1540 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca2fe4531954caba51132a547d82bfd (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260927 19:37:03.942432 1617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 960861c418c145508e7dc07cc2e571c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.943001 1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca2fe4531954caba51132a547d82bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.944449 1543 tablet_service.cc:1467] Processing CreateTablet for tablet 960861c418c145508e7dc07cc2e571c3 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260927 19:37:03.945993 1543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 960861c418c145508e7dc07cc2e571c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.946789 1615 tablet_service.cc:1467] Processing CreateTablet for tablet 69e273a93f5b488c98762a6950a85486 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260927 19:37:03.948513 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e273a93f5b488c98762a6950a85486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.950717 1614 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca2fe4531954caba51132a547d82bfd (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260927 19:37:03.952157 1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca2fe4531954caba51132a547d82bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.951900 1542 tablet_service.cc:1467] Processing CreateTablet for tablet e893d17f33d9498cb4d31f52016ab66c (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260927 19:37:03.950707 1541 tablet_service.cc:1467] Processing CreateTablet for tablet 69e273a93f5b488c98762a6950a85486 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260927 19:37:03.942513 1544 tablet_service.cc:1467] Processing CreateTablet for tablet cc5273de00ed4b70b5ee53cabc03c689 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260927 19:37:03.954972 1616 tablet_service.cc:1467] Processing CreateTablet for tablet e893d17f33d9498cb4d31f52016ab66c (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260927 19:37:03.957043 1616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893d17f33d9498cb4d31f52016ab66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.957561 1542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893d17f33d9498cb4d31f52016ab66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.961978 1669 tablet_bootstrap.cc:492] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:03.965781 1544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5273de00ed4b70b5ee53cabc03c689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.967851 1541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e273a93f5b488c98762a6950a85486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.968712 1669 tablet_bootstrap.cc:654] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:03.988288 1539 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc94069046c49f2b373529add98b664 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260927 19:37:03.988773 1619 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc94069046c49f2b373529add98b664 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260927 19:37:03.991340 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc94069046c49f2b373529add98b664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.992570 1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc94069046c49f2b373529add98b664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.006559 1467 tablet_service.cc:1467] Processing CreateTablet for tablet e893d17f33d9498cb4d31f52016ab66c (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260927 19:37:04.008040 1467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e893d17f33d9498cb4d31f52016ab66c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:03.999003 1538 tablet_service.cc:1467] Processing CreateTablet for tablet 82cd60fd586944ea9949e6cbd910624c (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260927 19:37:04.009938 1466 tablet_service.cc:1467] Processing CreateTablet for tablet 69e273a93f5b488c98762a6950a85486 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260927 19:37:04.011220 1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cd60fd586944ea9949e6cbd910624c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.011277 1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e273a93f5b488c98762a6950a85486. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.014945 1468 tablet_service.cc:1467] Processing CreateTablet for tablet 960861c418c145508e7dc07cc2e571c3 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260927 19:37:04.016358 1468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 960861c418c145508e7dc07cc2e571c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.000381 1470 tablet_service.cc:1467] Processing CreateTablet for tablet a89edbac45f14ad399aafe8ea1932989 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260927 19:37:04.019325 1465 tablet_service.cc:1467] Processing CreateTablet for tablet 4ca2fe4531954caba51132a547d82bfd (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260927 19:37:04.022634 1470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89edbac45f14ad399aafe8ea1932989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.022509 1464 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc94069046c49f2b373529add98b664 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260927 19:37:04.023985 1464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc94069046c49f2b373529add98b664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.025728 1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ca2fe4531954caba51132a547d82bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.026886 1463 tablet_service.cc:1467] Processing CreateTablet for tablet 82cd60fd586944ea9949e6cbd910624c (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260927 19:37:04.028450 1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cd60fd586944ea9949e6cbd910624c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.001138 1671 tablet_bootstrap.cc:492] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.002502 1469 tablet_service.cc:1467] Processing CreateTablet for tablet cc5273de00ed4b70b5ee53cabc03c689 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260927 19:37:04.054976 1469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5273de00ed4b70b5ee53cabc03c689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.069013 1619 tablet_service.cc:1467] Processing CreateTablet for tablet 82cd60fd586944ea9949e6cbd910624c (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260927 19:37:04.070992 1619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cd60fd586944ea9949e6cbd910624c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.079517 1465 tablet_service.cc:1467] Processing CreateTablet for tablet 5165cd333794441ab37c5bd95004b9a1 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260927 19:37:04.080788 1669 tablet_bootstrap.cc:492] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.081234 1669 ts_tablet_manager.cc:1397] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.120s user 0.009s sys 0.024s
I20260927 19:37:04.081425 1465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5165cd333794441ab37c5bd95004b9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.084291 1669 raft_consensus.cc:348] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.085052 1669 raft_consensus.cc:374] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.085340 1669 raft_consensus.cc:729] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.086477 1669 consensus_queue.cc:260] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.088498 1539 tablet_service.cc:1467] Processing CreateTablet for tablet 5165cd333794441ab37c5bd95004b9a1 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260927 19:37:04.089838 1539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5165cd333794441ab37c5bd95004b9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.091988 1614 tablet_service.cc:1467] Processing CreateTablet for tablet 5165cd333794441ab37c5bd95004b9a1 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260927 19:37:04.093242 1614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5165cd333794441ab37c5bd95004b9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.100808 1466 tablet_service.cc:1467] Processing CreateTablet for tablet ba423fb866274e838a9389da2ed20677 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260927 19:37:04.107815 1466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba423fb866274e838a9389da2ed20677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.107632 1615 tablet_service.cc:1467] Processing CreateTablet for tablet ba423fb866274e838a9389da2ed20677 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260927 19:37:04.117426 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba423fb866274e838a9389da2ed20677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.120599 1544 tablet_service.cc:1467] Processing CreateTablet for tablet ba423fb866274e838a9389da2ed20677 (DEFAULT_TABLE table=test-workload [id=1a7bd210264a4cf9ade57659803ba83f]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260927 19:37:04.129094 1669 ts_tablet_manager.cc:1428] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.047s user 0.007s sys 0.011s
I20260927 19:37:04.130749 1671 tablet_bootstrap.cc:654] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.140933 1654 heartbeater.cc:503] Master 127.0.204.190:36435 was elected leader, sending a full tablet report...
I20260927 19:37:04.146518 1669 tablet_bootstrap.cc:492] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.150168 1672 tablet_bootstrap.cc:492] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.161516 1544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba423fb866274e838a9389da2ed20677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:04.167316 1672 tablet_bootstrap.cc:654] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.169612 1669 tablet_bootstrap.cc:654] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.176607 1671 tablet_bootstrap.cc:492] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.177474 1671 ts_tablet_manager.cc:1397] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.177s user 0.029s sys 0.018s
I20260927 19:37:04.179425 1671 raft_consensus.cc:348] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.179994 1671 raft_consensus.cc:374] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.180274 1671 raft_consensus.cc:729] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.180928 1671 consensus_queue.cc:260] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.187476 1669 tablet_bootstrap.cc:492] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.188045 1669 ts_tablet_manager.cc:1397] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.010s
I20260927 19:37:04.191418 1580 heartbeater.cc:503] Master 127.0.204.190:36435 was elected leader, sending a full tablet report...
I20260927 19:37:04.191824 1671 ts_tablet_manager.cc:1428] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260927 19:37:04.191311 1669 raft_consensus.cc:348] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.192000 1672 tablet_bootstrap.cc:492] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.192291 1669 raft_consensus.cc:374] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.192701 1672 ts_tablet_manager.cc:1397] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.010s
I20260927 19:37:04.192755 1671 tablet_bootstrap.cc:492] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.192663 1669 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.194118 1669 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.195824 1672 raft_consensus.cc:348] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.197721 1672 raft_consensus.cc:374] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.198016 1672 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.198757 1672 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.199925 1671 tablet_bootstrap.cc:654] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.203007 1669 ts_tablet_manager.cc:1428] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.015s user 0.006s sys 0.003s
I20260927 19:37:04.204406 1669 tablet_bootstrap.cc:492] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.210548 1669 tablet_bootstrap.cc:654] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.211889 1505 heartbeater.cc:503] Master 127.0.204.190:36435 was elected leader, sending a full tablet report...
I20260927 19:37:04.212744 1672 ts_tablet_manager.cc:1428] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.019s user 0.003s sys 0.014s
I20260927 19:37:04.213701 1672 tablet_bootstrap.cc:492] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.220017 1672 tablet_bootstrap.cc:654] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.230451 1669 tablet_bootstrap.cc:492] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.231067 1671 tablet_bootstrap.cc:492] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.231200 1669 ts_tablet_manager.cc:1397] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.006s
I20260927 19:37:04.231665 1671 ts_tablet_manager.cc:1397] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.014s
I20260927 19:37:04.234555 1671 raft_consensus.cc:348] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.235448 1671 raft_consensus.cc:374] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.237112 1671 raft_consensus.cc:729] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.235237 1669 raft_consensus.cc:348] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.237908 1671 consensus_queue.cc:260] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.238575 1672 tablet_bootstrap.cc:492] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.238435 1669 raft_consensus.cc:374] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.239084 1672 ts_tablet_manager.cc:1397] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.009s
I20260927 19:37:04.238934 1669 raft_consensus.cc:729] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.240418 1671 ts_tablet_manager.cc:1428] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 19:37:04.240101 1669 consensus_queue.cc:260] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.241626 1671 tablet_bootstrap.cc:492] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.242228 1672 raft_consensus.cc:348] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.242959 1672 raft_consensus.cc:374] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.243218 1672 raft_consensus.cc:729] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.243899 1672 consensus_queue.cc:260] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.246395 1672 ts_tablet_manager.cc:1428] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:04.247296 1672 tablet_bootstrap.cc:492] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.248726 1671 tablet_bootstrap.cc:654] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.253944 1672 tablet_bootstrap.cc:654] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.258085 1673 raft_consensus.cc:482] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.258635 1673 raft_consensus.cc:504] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.261505 1671 tablet_bootstrap.cc:492] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.262039 1671 ts_tablet_manager.cc:1397] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20260927 19:37:04.262979 1673 leader_election.cc:288] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:04.244374 1669 ts_tablet_manager.cc:1428] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260927 19:37:04.264549 1669 tablet_bootstrap.cc:492] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.266074 1671 raft_consensus.cc:348] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.266906 1671 raft_consensus.cc:374] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.267288 1672 tablet_bootstrap.cc:492] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.267243 1671 raft_consensus.cc:729] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.267863 1672 ts_tablet_manager.cc:1397] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.006s
I20260927 19:37:04.268201 1671 consensus_queue.cc:260] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.270530 1671 ts_tablet_manager.cc:1428] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 19:37:04.270711 1672 raft_consensus.cc:348] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.271564 1671 tablet_bootstrap.cc:492] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.271485 1672 raft_consensus.cc:374] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.272078 1672 raft_consensus.cc:729] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.273058 1669 tablet_bootstrap.cc:654] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.272876 1672 consensus_queue.cc:260] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.275400 1672 ts_tablet_manager.cc:1428] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260927 19:37:04.276577 1672 tablet_bootstrap.cc:492] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.278083 1671 tablet_bootstrap.cc:654] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.284667 1677 raft_consensus.cc:482] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.285142 1672 tablet_bootstrap.cc:654] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.285243 1677 raft_consensus.cc:504] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.294294 1677 leader_election.cc:288] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.299919 1672 tablet_bootstrap.cc:492] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.300614 1672 ts_tablet_manager.cc:1397] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.005s
I20260927 19:37:04.302758 1669 tablet_bootstrap.cc:492] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.303237 1669 ts_tablet_manager.cc:1397] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.009s
I20260927 19:37:04.303427 1672 raft_consensus.cc:348] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.304211 1672 raft_consensus.cc:374] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.304490 1672 raft_consensus.cc:729] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.309542 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.310390 1480 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 0.
I20260927 19:37:04.310104 1669 raft_consensus.cc:348] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.312801 1669 raft_consensus.cc:374] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.313097 1592 leader_election.cc:302] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:04.313498 1669 raft_consensus.cc:729] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.314110 1673 raft_consensus.cc:2781] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.314532 1673 raft_consensus.cc:482] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.314318 1669 consensus_queue.cc:260] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.314862 1673 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.315662 1672 consensus_queue.cc:260] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.319340 1672 ts_tablet_manager.cc:1428] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.018s user 0.005s sys 0.005s
I20260927 19:37:04.322283 1672 tablet_bootstrap.cc:492] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.326725 1671 tablet_bootstrap.cc:492] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.327185 1671 ts_tablet_manager.cc:1397] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.007s
I20260927 19:37:04.329290 1673 raft_consensus.cc:504] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.329479 1671 raft_consensus.cc:348] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.330256 1671 raft_consensus.cc:374] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.330624 1671 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.331466 1671 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.334203 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.334777 1480 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.335688 1671 ts_tablet_manager.cc:1428] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 19:37:04.336688 1671 tablet_bootstrap.cc:492] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.338474 1672 tablet_bootstrap.cc:654] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.341547 1673 leader_election.cc:288] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:04.351096 1480 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 1.
I20260927 19:37:04.352406 1592 leader_election.cc:302] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:04.353384 1673 raft_consensus.cc:2781] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.353896 1673 raft_consensus.cc:686] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 LEADER]: Becoming Leader. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:04.354818 1673 consensus_queue.cc:237] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.356534 1677 raft_consensus.cc:482] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.358155 1677 raft_consensus.cc:504] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.361202 1677 leader_election.cc:288] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.367230 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.368248 1555 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 0.
I20260927 19:37:04.369954 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:04.370838 1554 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.367807 1671 tablet_bootstrap.cc:654] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.344594 1692 raft_consensus.cc:482] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.382934 1692 raft_consensus.cc:504] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.387840 1554 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 1.
I20260927 19:37:04.392447 1669 ts_tablet_manager.cc:1428] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.089s user 0.008s sys 0.046s
I20260927 19:37:04.393627 1669 tablet_bootstrap.cc:492] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.394904 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.395838 1692 leader_election.cc:288] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:04.396670 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.397482 1479 raft_consensus.cc:2454] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
W20260927 19:37:04.398991 1518 leader_election.cc:341] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.399730 1518 leader_election.cc:302] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.400121 1669 tablet_bootstrap.cc:654] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.400574 1677 raft_consensus.cc:2781] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.400951 1677 raft_consensus.cc:482] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.400883 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.401453 1677 raft_consensus.cc:3037] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.402562 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.400007 1672 tablet_bootstrap.cc:492] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.404400 1672 ts_tablet_manager.cc:1397] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.082s user 0.015s sys 0.030s
W20260927 19:37:04.403744 1592 leader_election.cc:341] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:04.405097 1593 leader_election.cc:341] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.402280 1375 catalog_manager.cc:5495] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a8943ab8566448487437f6561e08900 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "2a8943ab8566448487437f6561e08900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:04.408942 1672 raft_consensus.cc:348] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.409924 1672 raft_consensus.cc:374] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.410259 1672 raft_consensus.cc:729] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.411059 1672 consensus_queue.cc:260] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.412992 1677 raft_consensus.cc:504] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.414440 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.415146 1629 raft_consensus.cc:2454] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.415763 1671 tablet_bootstrap.cc:492] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.415839 1672 ts_tablet_manager.cc:1428] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260927 19:37:04.416219 1671 ts_tablet_manager.cc:1397] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.005s
I20260927 19:37:04.416344 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.417088 1672 tablet_bootstrap.cc:492] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.418185 1521 leader_election.cc:302] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters: 2703f921034946859b3f26666c525028
I20260927 19:37:04.419776 1593 leader_election.cc:302] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a8943ab8566448487437f6561e08900; no voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14
W20260927 19:37:04.420823 1426 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:04.422227 1677 leader_election.cc:288] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.422856 1677 raft_consensus.cc:2781] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.423194 1677 raft_consensus.cc:482] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.423629 1677 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.424470 1672 tablet_bootstrap.cc:654] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.426661 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.427309 1671 raft_consensus.cc:348] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.428474 1671 raft_consensus.cc:374] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.428874 1480 raft_consensus.cc:3037] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.435263 1677 raft_consensus.cc:504] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.434391 1669 tablet_bootstrap.cc:492] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.437412 1669 ts_tablet_manager.cc:1397] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.000s
I20260927 19:37:04.439069 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.440426 1692 raft_consensus.cc:2726] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 19:37:04.440927 1518 leader_election.cc:341] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.442229 1677 leader_election.cc:288] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.442292 1669 raft_consensus.cc:348] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.444823 1669 raft_consensus.cc:374] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.445741 1669 raft_consensus.cc:729] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.447154 1669 consensus_queue.cc:260] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
W20260927 19:37:04.447577 1521 leader_election.cc:341] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.444096 1480 raft_consensus.cc:2454] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.449198 1518 leader_election.cc:302] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.449905 1669 ts_tablet_manager.cc:1428] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260927 19:37:04.450027 1677 raft_consensus.cc:2781] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.449891 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.450640 1677 raft_consensus.cc:686] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.450708 1628 raft_consensus.cc:3037] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.451193 1669 tablet_bootstrap.cc:492] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.451918 1677 consensus_queue.cc:237] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.458047 1669 tablet_bootstrap.cc:654] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.459658 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.460300 1629 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.461529 1697 raft_consensus.cc:482] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.462253 1697 raft_consensus.cc:504] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.463567 1628 raft_consensus.cc:2454] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.466725 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.467466 1627 raft_consensus.cc:2454] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.428841 1671 raft_consensus.cc:729] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.468683 1671 consensus_queue.cc:260] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.469933 1672 tablet_bootstrap.cc:492] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.470407 1672 ts_tablet_manager.cc:1397] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.054s user 0.016s sys 0.027s
I20260927 19:37:04.471818 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.472453 1480 raft_consensus.cc:2454] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.473055 1672 raft_consensus.cc:348] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.473627 1518 leader_election.cc:302] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.473815 1672 raft_consensus.cc:374] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.474234 1672 raft_consensus.cc:729] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.474499 1677 raft_consensus.cc:2781] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.474865 1677 raft_consensus.cc:482] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.475195 1677 raft_consensus.cc:3037] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.475062 1672 consensus_queue.cc:260] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.479355 1375 catalog_manager.cc:5495] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.482331 1677 raft_consensus.cc:504] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.485260 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.485738 1672 ts_tablet_manager.cc:1428] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260927 19:37:04.485630 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.485900 1480 raft_consensus.cc:3037] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.486243 1627 raft_consensus.cc:3037] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.492535 1672 tablet_bootstrap.cc:492] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.493125 1697 leader_election.cc:288] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223), 2703f921034946859b3f26666c525028 (127.0.204.129:45337)
I20260927 19:37:04.492648 1480 raft_consensus.cc:2454] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.494511 1678 raft_consensus.cc:482] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.494993 1678 raft_consensus.cc:504] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.499768 1672 tablet_bootstrap.cc:654] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.505450 1678 leader_election.cc:288] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.512501 1672 tablet_bootstrap.cc:492] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.513058 1672 ts_tablet_manager.cc:1397] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.003s
I20260927 19:37:04.513665 1629 raft_consensus.cc:2454] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.517442 1521 leader_election.cc:302] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters: 2703f921034946859b3f26666c525028
I20260927 19:37:04.517503 1672 raft_consensus.cc:348] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.518774 1672 raft_consensus.cc:374] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.519212 1672 raft_consensus.cc:729] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.519279 1697 raft_consensus.cc:2781] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.520020 1697 raft_consensus.cc:686] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.520222 1672 consensus_queue.cc:260] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.520617 1518 leader_election.cc:302] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.521075 1697 consensus_queue.cc:237] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.509131 1627 raft_consensus.cc:2454] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.525828 1677 leader_election.cc:288] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223), 2703f921034946859b3f26666c525028 (127.0.204.129:45337)
I20260927 19:37:04.533816 1677 raft_consensus.cc:2781] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.534407 1677 raft_consensus.cc:686] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.535146 1677 consensus_queue.cc:237] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:04.536108 1708 raft_consensus.cc:482] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.537295 1672 ts_tablet_manager.cc:1428] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.024s user 0.007s sys 0.001s
I20260927 19:37:04.538582 1708 raft_consensus.cc:504] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.543052 1378 catalog_manager.cc:5495] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.544941 1708 leader_election.cc:288] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.541539 1671 ts_tablet_manager.cc:1428] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.125s user 0.011s sys 0.025s
I20260927 19:37:04.546584 1671 tablet_bootstrap.cc:492] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.546381 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.546378 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.547149 1480 raft_consensus.cc:2454] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
W20260927 19:37:04.547999 1521 leader_election.cc:341] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.548614 1518 leader_election.cc:302] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters: 2a8943ab8566448487437f6561e08900
I20260927 19:37:04.549887 1708 raft_consensus.cc:2781] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.550241 1708 raft_consensus.cc:482] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.550565 1708 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.542511 1672 tablet_bootstrap.cc:492] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.554795 1671 tablet_bootstrap.cc:654] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.539842 1669 tablet_bootstrap.cc:492] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.560948 1669 ts_tablet_manager.cc:1397] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.110s user 0.018s sys 0.023s
I20260927 19:37:04.564108 1669 raft_consensus.cc:348] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.564271 1708 raft_consensus.cc:504] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.565004 1669 raft_consensus.cc:374] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.565407 1669 raft_consensus.cc:729] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.566256 1669 consensus_queue.cc:260] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.569087 1669 ts_tablet_manager.cc:1428] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 19:37:04.574878 1669 tablet_bootstrap.cc:492] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.576180 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.577016 1480 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.582584 1669 tablet_bootstrap.cc:654] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.586732 1480 raft_consensus.cc:2454] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.574913 1672 tablet_bootstrap.cc:654] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.590428 1518 leader_election.cc:302] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.590641 1708 leader_election.cc:288] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.591586 1708 raft_consensus.cc:2781] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.592871 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
W20260927 19:37:04.594357 1521 leader_election.cc:341] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.598290 1708 raft_consensus.cc:686] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.599915 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.600528 1708 consensus_queue.cc:237] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
W20260927 19:37:04.601668 1444 leader_election.cc:341] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.603984 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.605819 1627 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 0.
I20260927 19:37:04.607013 1671 tablet_bootstrap.cc:492] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.607142 1446 leader_election.cc:302] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters: 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:04.607568 1671 ts_tablet_manager.cc:1397] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.026s
I20260927 19:37:04.608747 1678 raft_consensus.cc:2781] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.609123 1678 raft_consensus.cc:482] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.603895 1378 catalog_manager.cc:5495] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.611323 1671 raft_consensus.cc:348] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.611848 1669 tablet_bootstrap.cc:492] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.611675 1678 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.612397 1671 raft_consensus.cc:374] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.612797 1669 ts_tablet_manager.cc:1397] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.038s user 0.018s sys 0.001s
I20260927 19:37:04.612740 1671 raft_consensus.cc:729] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.613765 1671 consensus_queue.cc:260] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.616446 1671 ts_tablet_manager.cc:1428] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260927 19:37:04.616268 1669 raft_consensus.cc:348] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.617281 1669 raft_consensus.cc:374] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.617481 1671 tablet_bootstrap.cc:492] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.617735 1669 raft_consensus.cc:729] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.618673 1669 consensus_queue.cc:260] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.623646 1669 ts_tablet_manager.cc:1428] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260927 19:37:04.624267 1671 tablet_bootstrap.cc:654] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.624856 1669 tablet_bootstrap.cc:492] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.633515 1669 tablet_bootstrap.cc:654] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.661775 1678 raft_consensus.cc:504] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.665323 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
W20260927 19:37:04.666894 1444 leader_election.cc:341] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.669040 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.669756 1627 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.672931 1672 tablet_bootstrap.cc:492] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.673444 1672 ts_tablet_manager.cc:1397] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.135s user 0.006s sys 0.009s
I20260927 19:37:04.673489 1671 tablet_bootstrap.cc:492] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.674026 1671 ts_tablet_manager.cc:1397] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.003s
I20260927 19:37:04.674825 1669 tablet_bootstrap.cc:492] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.675366 1669 ts_tablet_manager.cc:1397] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.017s
I20260927 19:37:04.676268 1672 raft_consensus.cc:348] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.676990 1672 raft_consensus.cc:374] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.677435 1672 raft_consensus.cc:729] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.677655 1671 raft_consensus.cc:348] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.678395 1671 raft_consensus.cc:374] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.678161 1672 consensus_queue.cc:260] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.678756 1671 raft_consensus.cc:729] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.678275 1669 raft_consensus.cc:348] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.679389 1669 raft_consensus.cc:374] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.679695 1669 raft_consensus.cc:729] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.679725 1671 consensus_queue.cc:260] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.680590 1669 consensus_queue.cc:260] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.682140 1671 ts_tablet_manager.cc:1428] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 19:37:04.683094 1669 ts_tablet_manager.cc:1428] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260927 19:37:04.683161 1671 tablet_bootstrap.cc:492] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.684141 1669 tablet_bootstrap.cc:492] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.689693 1669 tablet_bootstrap.cc:654] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.689697 1671 tablet_bootstrap.cc:654] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.701824 1669 tablet_bootstrap.cc:492] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.702291 1669 ts_tablet_manager.cc:1397] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.013s
I20260927 19:37:04.702742 1671 tablet_bootstrap.cc:492] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.703274 1671 ts_tablet_manager.cc:1397] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.007s
I20260927 19:37:04.704766 1669 raft_consensus.cc:348] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.705694 1669 raft_consensus.cc:374] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.705979 1669 raft_consensus.cc:729] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.706207 1671 raft_consensus.cc:348] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.707089 1671 raft_consensus.cc:374] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.707449 1671 raft_consensus.cc:729] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.676978 1627 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 1.
I20260927 19:37:04.708838 1671 consensus_queue.cc:260] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.709649 1446 leader_election.cc:302] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters: 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:04.711573 1678 leader_election.cc:288] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.680646 1375 catalog_manager.cc:5495] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.712502 1671 ts_tablet_manager.cc:1428] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260927 19:37:04.713923 1671 tablet_bootstrap.cc:492] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.716619 1678 raft_consensus.cc:2781] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.717263 1678 raft_consensus.cc:686] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:04.723321 1708 raft_consensus.cc:482] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.723941 1708 raft_consensus.cc:504] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.726179 1708 leader_election.cc:288] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.727283 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.727849 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.732807 1671 tablet_bootstrap.cc:654] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.707424 1669 consensus_queue.cc:260] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.720495 1678 consensus_queue.cc:237] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.734470 1627 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.737548 1521 leader_election.cc:302] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.738984 1708 raft_consensus.cc:2781] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.739408 1708 raft_consensus.cc:482] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.739728 1708 raft_consensus.cc:3037] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.740742 1669 ts_tablet_manager.cc:1428] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.038s user 0.004s sys 0.003s
I20260927 19:37:04.741699 1669 tablet_bootstrap.cc:492] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900: Bootstrap starting.
I20260927 19:37:04.729321 1480 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.747761 1717 raft_consensus.cc:482] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.748999 1692 raft_consensus.cc:482] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.750648 1692 raft_consensus.cc:504] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.750771 1717 raft_consensus.cc:504] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.754586 1673 raft_consensus.cc:482] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.754928 1669 tablet_bootstrap.cc:654] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.755154 1673 raft_consensus.cc:504] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.757292 1672 ts_tablet_manager.cc:1428] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.083s user 0.024s sys 0.027s
I20260927 19:37:04.757422 1673 leader_election.cc:288] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:04.758432 1672 tablet_bootstrap.cc:492] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.758582 1692 leader_election.cc:288] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:04.761977 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.762081 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.762729 1669 tablet_bootstrap.cc:492] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900: No bootstrap required, opened a new log
I20260927 19:37:04.763163 1669 ts_tablet_manager.cc:1397] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.005s
I20260927 19:37:04.764391 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.764935 1669 raft_consensus.cc:348] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.765473 1669 raft_consensus.cc:374] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.765692 1669 raft_consensus.cc:729] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.765770 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.765654 1708 raft_consensus.cc:504] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.766332 1669 consensus_queue.cc:260] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.766731 1479 raft_consensus.cc:2379] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a8943ab8566448487437f6561e08900 in current term 1: Already voted for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in this term.
I20260927 19:37:04.767092 1554 raft_consensus.cc:2379] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2703f921034946859b3f26666c525028 in current term 1: Already voted for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in this term.
W20260927 19:37:04.768615 1593 leader_election.cc:341] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.768918 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.769626 1479 raft_consensus.cc:3037] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:04.770053 1592 leader_election.cc:341] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.770500 1592 leader_election.cc:302] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a8943ab8566448487437f6561e08900; no voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:04.771811 1717 leader_election.cc:288] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.771857 1692 raft_consensus.cc:2726] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:04.772930 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.774222 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.775266 1627 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 0.
I20260927 19:37:04.776814 1593 leader_election.cc:302] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a8943ab8566448487437f6561e08900; no voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:04.777863 1446 leader_election.cc:302] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:04.779045 1692 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.780088 1669 ts_tablet_manager.cc:1428] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260927 19:37:04.783277 1671 tablet_bootstrap.cc:492] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.784080 1671 ts_tablet_manager.cc:1397] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.070s user 0.013s sys 0.008s
I20260927 19:37:04.785657 1692 raft_consensus.cc:2726] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:04.787333 1708 leader_election.cc:288] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.789821 1479 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.790715 1717 raft_consensus.cc:2741] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 19:37:04.792713 1518 leader_election.cc:302] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.793799 1708 raft_consensus.cc:2781] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.793094 1377 catalog_manager.cc:5495] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.795114 1708 raft_consensus.cc:686] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.795843 1671 raft_consensus.cc:348] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.796654 1671 raft_consensus.cc:374] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.796954 1671 raft_consensus.cc:729] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.797778 1672 tablet_bootstrap.cc:654] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.797464 1708 consensus_queue.cc:237] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.798731 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.798851 1671 consensus_queue.cc:260] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.799357 1627 raft_consensus.cc:3037] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.801420 1704 raft_consensus.cc:482] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.802135 1671 ts_tablet_manager.cc:1428] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260927 19:37:04.802336 1704 raft_consensus.cc:504] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.803164 1671 tablet_bootstrap.cc:492] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: Bootstrap starting.
I20260927 19:37:04.804718 1704 leader_election.cc:288] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.806924 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.808774 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.810320 1629 raft_consensus.cc:2379] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e54647c35ee47b48d8c7a4bf445ba14 in current term 1: Already voted for candidate 2703f921034946859b3f26666c525028 in this term.
I20260927 19:37:04.811470 1521 leader_election.cc:302] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e54647c35ee47b48d8c7a4bf445ba14; no voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900
I20260927 19:37:04.812234 1704 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.813309 1671 tablet_bootstrap.cc:654] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.811988 1672 tablet_bootstrap.cc:492] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.814283 1672 ts_tablet_manager.cc:1397] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.007s
I20260927 19:37:04.814872 1677 raft_consensus.cc:482] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.815312 1677 raft_consensus.cc:504] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.816727 1672 raft_consensus.cc:348] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.817466 1672 raft_consensus.cc:374] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.817741 1672 raft_consensus.cc:729] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.818377 1672 consensus_queue.cc:260] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.817195 1677 leader_election.cc:288] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.819023 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.819618 1629 raft_consensus.cc:2454] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.820827 1672 ts_tablet_manager.cc:1428] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 19:37:04.820935 1521 leader_election.cc:302] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.819641 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.821766 1672 tablet_bootstrap.cc:492] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028: Bootstrap starting.
I20260927 19:37:04.821836 1677 raft_consensus.cc:2781] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.822279 1677 raft_consensus.cc:482] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:37:04.822309 1518 leader_election.cc:341] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.822714 1677 raft_consensus.cc:3037] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.824075 1704 raft_consensus.cc:2726] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:04.827536 1672 tablet_bootstrap.cc:654] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:04.827273 1627 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.835645 1677 raft_consensus.cc:504] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.844023 1677 leader_election.cc:288] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.846148 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
W20260927 19:37:04.847630 1518 leader_election.cc:341] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:04.848081 1717 raft_consensus.cc:482] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.848515 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.848575 1717 raft_consensus.cc:504] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.849273 1627 raft_consensus.cc:3037] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.851584 1717 leader_election.cc:288] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.853698 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.854592 1629 raft_consensus.cc:2379] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2703f921034946859b3f26666c525028 in current term 1: Already voted for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in this term.
I20260927 19:37:04.855726 1671 tablet_bootstrap.cc:492] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: No bootstrap required, opened a new log
I20260927 19:37:04.856267 1671 ts_tablet_manager.cc:1397] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent bootstrapping tablet: real 0.053s user 0.021s sys 0.016s
I20260927 19:37:04.857117 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:04.858858 1444 leader_election.cc:302] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2703f921034946859b3f26666c525028; no voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:04.859105 1671 raft_consensus.cc:348] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.860119 1678 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.860110 1671 raft_consensus.cc:374] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.861680 1671 raft_consensus.cc:729] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.862449 1671 consensus_queue.cc:260] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.863636 1377 catalog_manager.cc:5495] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.864694 1671 ts_tablet_manager.cc:1428] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 19:37:04.869459 1672 tablet_bootstrap.cc:492] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028: No bootstrap required, opened a new log
I20260927 19:37:04.870018 1672 ts_tablet_manager.cc:1397] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.010s
I20260927 19:37:04.870949 1627 raft_consensus.cc:2454] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.872047 1521 leader_election.cc:302] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters: 2703f921034946859b3f26666c525028
I20260927 19:37:04.872530 1672 raft_consensus.cc:348] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.873085 1672 raft_consensus.cc:374] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:04.873152 1708 raft_consensus.cc:2781] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.873330 1672 raft_consensus.cc:729] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Initialized, Role: FOLLOWER
I20260927 19:37:04.873711 1708 raft_consensus.cc:686] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.874080 1672 consensus_queue.cc:260] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.874611 1678 raft_consensus.cc:2726] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:04.876168 1672 ts_tablet_manager.cc:1428] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260927 19:37:04.874554 1708 consensus_queue.cc:237] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.886214 1377 catalog_manager.cc:5495] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:04.919770 1692 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:04.930779 1708 raft_consensus.cc:482] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:04.931278 1708 raft_consensus.cc:504] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.933081 1708 leader_election.cc:288] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.933772 1732 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:04.934643 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba423fb866274e838a9389da2ed20677" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:04.936074 1627 raft_consensus.cc:2454] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.937101 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba423fb866274e838a9389da2ed20677" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:04.937218 1521 leader_election.cc:302] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.937788 1479 raft_consensus.cc:2454] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:04.938295 1677 raft_consensus.cc:2781] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:04.938755 1677 raft_consensus.cc:482] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:04.939164 1677 raft_consensus.cc:3037] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:04.941303 1581 tablet.cc:2369] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 19:37:04.946856 1704 consensus_queue.cc:1035] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:04.949962 1677 raft_consensus.cc:504] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.952739 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba423fb866274e838a9389da2ed20677" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:04.953106 1677 leader_election.cc:288] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:04.953271 1479 raft_consensus.cc:3037] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.953389 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba423fb866274e838a9389da2ed20677" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:04.954010 1629 raft_consensus.cc:3037] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:04.960134 1479 raft_consensus.cc:2454] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.960230 1629 raft_consensus.cc:2454] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:04.961279 1518 leader_election.cc:302] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:04.962193 1697 raft_consensus.cc:2781] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:04.962628 1697 raft_consensus.cc:686] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:04.963410 1697 consensus_queue.cc:237] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:04.966296 1704 consensus_queue.cc:1035] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:04.982507 1378 catalog_manager.cc:5495] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:05.001219 1697 consensus_queue.cc:1035] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.012024 1704 consensus_queue.cc:1035] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.036043 1704 consensus_queue.cc:1035] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:05.036789 1677 consensus_queue.cc:1035] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.048990 1697 consensus_queue.cc:1035] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:05.054448 1743 consensus_queue.cc:1035] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:05.106262 1743 raft_consensus.cc:482] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:05.106755 1743 raft_consensus.cc:504] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.108301 1743 leader_election.cc:288] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:05.109475 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028" is_pre_election: true
I20260927 19:37:05.109476 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:05.110395 1480 raft_consensus.cc:2454] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:05.110373 1627 raft_consensus.cc:2454] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 0.
I20260927 19:37:05.111685 1518 leader_election.cc:302] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:05.112414 1743 raft_consensus.cc:2781] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:05.112772 1743 raft_consensus.cc:482] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:05.113067 1743 raft_consensus.cc:3037] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:05.118498 1743 raft_consensus.cc:504] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.120090 1743 leader_election.cc:288] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:05.120993 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:05.121061 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:05.121771 1480 raft_consensus.cc:3037] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:05.121801 1627 raft_consensus.cc:3037] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:05.126688 1627 raft_consensus.cc:2454] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:05.126802 1480 raft_consensus.cc:2454] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 1.
I20260927 19:37:05.127949 1521 leader_election.cc:302] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:05.128681 1743 raft_consensus.cc:2781] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:05.129244 1743 raft_consensus.cc:686] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:05.130211 1743 consensus_queue.cc:237] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.137197 1678 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.137524 1378 catalog_manager.cc:5495] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:05.148005 1721 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:05.321801 1743 consensus_queue.cc:1035] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.327816 1721 raft_consensus.cc:482] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:05.328387 1721 raft_consensus.cc:504] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.330432 1721 leader_election.cc:288] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:05.332279 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" is_pre_election: true
I20260927 19:37:05.332494 1697 consensus_queue.cc:1035] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.334154 1627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5165cd333794441ab37c5bd95004b9a1" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8943ab8566448487437f6561e08900" is_pre_election: true
I20260927 19:37:05.334929 1627 raft_consensus.cc:2379] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2703f921034946859b3f26666c525028 in current term 1: Already voted for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in this term.
I20260927 19:37:05.335897 1446 leader_election.cc:302] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2703f921034946859b3f26666c525028; no voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:05.337013 1721 raft_consensus.cc:3037] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:05.344116 1721 raft_consensus.cc:2726] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:05.348762 1697 consensus_queue.cc:1035] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.365442 1697 consensus_queue.cc:1035] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:05.385425 1554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:05.386092 1554 raft_consensus.cc:595] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:05.391373 1554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:05.391970 1554 raft_consensus.cc:595] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:05.406973 1629 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2a8943ab8566448487437f6561e08900"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44342
I20260927 19:37:05.407500 1629 raft_consensus.cc:595] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:05.412698 1554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:05.413305 1554 raft_consensus.cc:595] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:05.416569 1732 raft_consensus.cc:979] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.417040 1673 raft_consensus.cc:1067] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.418872 1479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47020
I20260927 19:37:05.419490 1479 raft_consensus.cc:482] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:05.419884 1479 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.421006 1697 consensus_queue.cc:1035] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.426323 1479 raft_consensus.cc:504] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.429543 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:05.429966 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:05.430215 1554 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.430511 1629 raft_consensus.cc:3032] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:05.430822 1629 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:05.431562 1629 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.432623 1629 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.436748 1554 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.439406 1629 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.440012 1479 leader_election.cc:288] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:05.440478 1446 leader_election.cc:302] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:05.441315 1721 raft_consensus.cc:2781] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:05.441766 1721 raft_consensus.cc:686] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:05.442561 1721 consensus_queue.cc:237] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.443622 1697 consensus_queue.cc:1035] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.455135 1375 catalog_manager.cc:5495] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 2a8943ab8566448487437f6561e08900 (127.0.204.131) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:05.458581 1554 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba423fb866274e838a9389da2ed20677"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:05.459242 1554 raft_consensus.cc:595] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2a8943ab8566448487437f6561e08900
I20260927 19:37:05.462248 1704 raft_consensus.cc:979] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:05.462677 1697 raft_consensus.cc:1067] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:05.464273 1629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba423fb866274e838a9389da2ed20677"
dest_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:44316
I20260927 19:37:05.464831 1629 raft_consensus.cc:482] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:05.465144 1629 raft_consensus.cc:3037] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.471707 1629 raft_consensus.cc:504] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.473385 1629 leader_election.cc:288] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 2 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:05.474370 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba423fb866274e838a9389da2ed20677" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:05.474409 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba423fb866274e838a9389da2ed20677" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:05.475069 1479 raft_consensus.cc:3037] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.475059 1554 raft_consensus.cc:3032] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:05.475589 1554 raft_consensus.cc:729] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:05.476361 1554 consensus_queue.cc:260] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.477632 1554 raft_consensus.cc:3037] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.481901 1479 raft_consensus.cc:2454] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 2.
I20260927 19:37:05.483249 1592 leader_election.cc:302] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:05.484019 1673 raft_consensus.cc:2781] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:05.484498 1673 raft_consensus.cc:686] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Becoming Leader. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:05.483994 1554 raft_consensus.cc:2454] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 2.
I20260927 19:37:05.485668 1697 raft_consensus.cc:979] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.485321 1673 consensus_queue.cc:237] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.486192 1708 raft_consensus.cc:1067] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.487748 1479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47030
I20260927 19:37:05.488302 1479 raft_consensus.cc:482] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:05.488690 1479 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.492188 1375 catalog_manager.cc:5495] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2a8943ab8566448487437f6561e08900 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "2a8943ab8566448487437f6561e08900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:05.493685 1479 raft_consensus.cc:504] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.495451 1479 leader_election.cc:288] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:05.496587 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:05.496573 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:05.497287 1554 raft_consensus.cc:3032] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:05.497378 1629 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.497725 1554 raft_consensus.cc:729] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:05.498565 1554 consensus_queue.cc:260] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.499784 1554 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.504256 1629 raft_consensus.cc:2454] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.505482 1446 leader_election.cc:302] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:05.506345 1554 raft_consensus.cc:2454] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.506378 1721 raft_consensus.cc:2781] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:05.506989 1721 raft_consensus.cc:686] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:05.507831 1721 consensus_queue.cc:237] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.514406 1375 catalog_manager.cc:5495] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:05.520622 1704 raft_consensus.cc:979] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.521073 1704 raft_consensus.cc:1067] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.522987 1479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47030
I20260927 19:37:05.523636 1479 raft_consensus.cc:482] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:05.524039 1479 raft_consensus.cc:3037] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.529603 1479 raft_consensus.cc:504] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:05.532429 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:05.532612 1479 leader_election.cc:288] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:05.533138 1629 raft_consensus.cc:3037] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.533612 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89edbac45f14ad399aafe8ea1932989" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:05.534183 1554 raft_consensus.cc:3032] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:05.534521 1554 raft_consensus.cc:729] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:05.535264 1554 consensus_queue.cc:260] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:05.536455 1554 raft_consensus.cc:3037] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.540259 1629 raft_consensus.cc:2454] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.541280 1446 leader_election.cc:302] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:05.542083 1721 raft_consensus.cc:2781] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:05.542462 1554 raft_consensus.cc:2454] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.542490 1721 raft_consensus.cc:686] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:05.543540 1721 consensus_queue.cc:237] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } }
I20260927 19:37:05.547676 1697 consensus_queue.cc:1035] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.550544 1374 catalog_manager.cc:5495] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:05.559650 1708 consensus_queue.cc:1035] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:05.566310 1704 raft_consensus.cc:979] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.566726 1708 raft_consensus.cc:1067] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:05.568149 1479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47030
I20260927 19:37:05.568784 1479 raft_consensus.cc:482] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:05.569214 1479 raft_consensus.cc:3037] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.575058 1479 raft_consensus.cc:504] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.577541 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:05.577873 1479 leader_election.cc:288] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:05.578166 1554 raft_consensus.cc:3032] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:05.578076 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5273de00ed4b70b5ee53cabc03c689" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:05.578569 1554 raft_consensus.cc:729] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:05.578755 1629 raft_consensus.cc:3037] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.579519 1554 consensus_queue.cc:260] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.580803 1554 raft_consensus.cc:3037] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:05.584237 1629 raft_consensus.cc:2454] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.585281 1446 leader_election.cc:302] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:05.586077 1554 raft_consensus.cc:2454] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:05.586130 1721 raft_consensus.cc:2781] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:05.586882 1721 raft_consensus.cc:686] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:05.587723 1721 consensus_queue.cc:237] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:05.594341 1377 catalog_manager.cc:5495] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:05.908991 1554 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.909340 1479 raft_consensus.cc:1261] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.910313 1760 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:05.910741 1673 consensus_queue.cc:1035] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:05.920732 1555 raft_consensus.cc:1261] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.922482 1732 consensus_queue.cc:1035] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:05.924305 1629 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.926149 1720 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:05.931306 1555 raft_consensus.cc:1261] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.933079 1721 consensus_queue.cc:1035] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:05.945048 1629 raft_consensus.cc:1261] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.946616 1760 consensus_queue.cc:1035] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:05.992137 1629 raft_consensus.cc:1261] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:05.993240 1760 consensus_queue.cc:1035] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:06.000991 1554 raft_consensus.cc:1261] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:06.003273 1720 consensus_queue.cc:1035] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:06.071053 1554 raft_consensus.cc:1261] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:06.072245 1720 consensus_queue.cc:1035] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:06.080847 1629 raft_consensus.cc:1261] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:06.082244 1721 consensus_queue.cc:1035] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:08.495136 818 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260927 19:37:08.496055 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:08.521602 1479 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69e273a93f5b488c98762a6950a85486"
dest_uuid: "2703f921034946859b3f26666c525028"
mode: GRACEFUL
new_leader_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:47044
I20260927 19:37:08.522182 1479 raft_consensus.cc:595] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 LEADER]: Received request to transfer leadership to TS 2a8943ab8566448487437f6561e08900
I20260927 19:37:08.524749 1479 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2703f921034946859b3f26666c525028"
mode: GRACEFUL
new_leader_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:47044
I20260927 19:37:08.525286 1479 raft_consensus.cc:595] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Received request to transfer leadership to TS 2a8943ab8566448487437f6561e08900
I20260927 19:37:08.526284 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260927 19:37:08.526808 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:08.531944 1760 raft_consensus.cc:979] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: : Instructing follower 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:08.532413 1760 raft_consensus.cc:1067] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Signalling peer 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:08.533995 1629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:44330
I20260927 19:37:08.534433 1629 raft_consensus.cc:482] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:08.534669 1629 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:08.539579 1629 raft_consensus.cc:504] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:08.541529 1629 leader_election.cc:288] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 3 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:08.541942 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:08.542133 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:08.542557 1479 raft_consensus.cc:3032] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:37:08.542642 1554 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:08.542955 1479 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:08.543713 1479 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:08.544907 1479 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:08.549449 1554 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 3.
I20260927 19:37:08.550791 1593 leader_election.cc:302] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:08.551494 1479 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 3.
I20260927 19:37:08.551631 1764 raft_consensus.cc:2781] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:37:08.552685 1764 raft_consensus.cc:686] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 LEADER]: Becoming Leader. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:08.553478 1764 consensus_queue.cc:237] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:08.560925 1377 catalog_manager.cc:5495] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 reported cstate change: term changed from 2 to 3, leader changed from 2703f921034946859b3f26666c525028 (127.0.204.129) to 2a8943ab8566448487437f6561e08900 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "2a8943ab8566448487437f6561e08900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:08.979974 1479 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:08.981266 1764 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:08.990856 1554 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:08.992372 1764 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:09.028569 1767 raft_consensus.cc:979] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: : Instructing follower 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:09.029034 1767 raft_consensus.cc:1067] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 LEADER]: Signalling peer 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:09.030490 1629 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69e273a93f5b488c98762a6950a85486"
dest_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:44330
I20260927 19:37:09.030946 1629 raft_consensus.cc:482] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:09.031221 1629 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:09.035260 1629 raft_consensus.cc:504] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:09.036583 1629 leader_election.cc:288] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 2 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:09.037778 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:09.037776 1479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:09.038446 1554 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:09.038486 1479 raft_consensus.cc:3032] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:09.039067 1479 raft_consensus.cc:729] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:09.040050 1479 consensus_queue.cc:260] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:09.041491 1479 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:09.046092 1554 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 2.
I20260927 19:37:09.047386 1593 leader_election.cc:302] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:09.048272 1764 raft_consensus.cc:2781] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:09.048331 1479 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 2.
I20260927 19:37:09.048806 1764 raft_consensus.cc:686] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Becoming Leader. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:09.049641 1764 consensus_queue.cc:237] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:09.055980 1377 catalog_manager.cc:5495] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 reported cstate change: term changed from 1 to 2, leader changed from 2703f921034946859b3f26666c525028 (127.0.204.129) to 2a8943ab8566448487437f6561e08900 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "2a8943ab8566448487437f6561e08900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:09.476778 1479 raft_consensus.cc:1261] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:09.478044 1764 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:09.488807 1554 raft_consensus.cc:1261] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:09.492214 1764 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:09.527673 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:09.533267 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:09.533777 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:10.534561 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:10.540933 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:10.541457 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:11.542299 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:11.548173 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:11.548590 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:12.549633 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:12.555055 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:12.555526 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:13.556373 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:13.561544 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:13.562014 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:15.258970 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:15.264065 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:15.264511 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260927 19:37:15.438867 1576 debug-util.cc:398] Leaking SignalData structure 0x7b08001049c0 after lost signal to thread 821
W20260927 19:37:15.439600 1576 debug-util.cc:398] Leaking SignalData structure 0x7b08001150a0 after lost signal to thread 1412
W20260927 19:37:15.440338 1576 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8020 after lost signal to thread 1504
W20260927 19:37:15.441041 1576 debug-util.cc:398] Leaking SignalData structure 0x7b08001bcee0 after lost signal to thread 1579
W20260927 19:37:15.441670 1576 debug-util.cc:398] Leaking SignalData structure 0x7b08001d1440 after lost signal to thread 1653
I20260927 19:37:16.266115 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:16.275213 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:16.275815 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:17.276909 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:17.285554 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:17.286149 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:18.287072 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:18.294080 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:18.294726 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:19.328346 818 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260927 19:37:19.337766 1628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2a8943ab8566448487437f6561e08900"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44342
I20260927 19:37:19.338610 1628 raft_consensus.cc:595] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:19.346472 1628 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69e273a93f5b488c98762a6950a85486"
dest_uuid: "2a8943ab8566448487437f6561e08900"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44342
I20260927 19:37:19.346979 1628 raft_consensus.cc:595] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:19.352080 1555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:19.352761 1555 raft_consensus.cc:595] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:19.358409 1555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0dc94069046c49f2b373529add98b664"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:19.358925 1555 raft_consensus.cc:595] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:19.363464 1555 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
mode: GRACEFUL
new_leader_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:44408
I20260927 19:37:19.364023 1555 raft_consensus.cc:595] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Received request to transfer leadership to TS 2703f921034946859b3f26666c525028
I20260927 19:37:19.492393 1793 raft_consensus.cc:979] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.492916 1800 raft_consensus.cc:1067] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.494472 1479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0dc94069046c49f2b373529add98b664"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47030
I20260927 19:37:19.494987 1479 raft_consensus.cc:482] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:19.495339 1479 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.501621 1479 raft_consensus.cc:504] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.504496 1479 leader_election.cc:288] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:19.505807 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:19.505779 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:19.506724 1555 raft_consensus.cc:3032] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:19.507019 1628 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.507066 1555 raft_consensus.cc:729] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:19.507719 1799 raft_consensus.cc:979] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.508189 1799 raft_consensus.cc:1067] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.508399 1800 raft_consensus.cc:979] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.508051 1555 consensus_queue.cc:260] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.508838 1800 raft_consensus.cc:1067] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.509801 1555 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.510409 1479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69e273a93f5b488c98762a6950a85486"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47020
I20260927 19:37:19.510927 1480 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47030
I20260927 19:37:19.510970 1479 raft_consensus.cc:482] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:19.511598 1480 raft_consensus.cc:482] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:19.511719 1479 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:19.512044 1480 raft_consensus.cc:3037] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.516659 1628 raft_consensus.cc:2454] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:19.517336 1555 raft_consensus.cc:2454] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:19.518148 1446 leader_election.cc:302] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:19.519568 1796 raft_consensus.cc:2781] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:19.519788 1479 raft_consensus.cc:504] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.520906 1796 raft_consensus.cc:686] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:19.521863 1796 consensus_queue.cc:237] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.522178 1480 raft_consensus.cc:504] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.522522 1479 leader_election.cc:288] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 3 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:19.522782 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:19.523473 1555 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:19.523756 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:19.524407 1628 raft_consensus.cc:3032] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:37:19.524729 1628 raft_consensus.cc:729] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:19.525280 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:19.525859 1554 raft_consensus.cc:3032] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:19.526121 1554 raft_consensus.cc:729] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:19.525503 1628 consensus_queue.cc:260] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.526916 1554 consensus_queue.cc:260] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.527503 1628 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:19.527942 1554 raft_consensus.cc:3037] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.530679 1555 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 3.
I20260927 19:37:19.531945 1480 leader_election.cc:288] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:19.532016 1444 leader_election.cc:302] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:19.533349 1801 raft_consensus.cc:2781] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:37:19.533551 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cd60fd586944ea9949e6cbd910624c" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:19.534071 1629 raft_consensus.cc:3037] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.538591 1628 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 3.
I20260927 19:37:19.540997 1554 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:19.542253 1444 leader_election.cc:302] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:19.546208 1629 raft_consensus.cc:2454] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:19.548365 1804 raft_consensus.cc:2781] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:19.548810 1801 raft_consensus.cc:686] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:19.549261 1804 raft_consensus.cc:686] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:19.549772 1801 consensus_queue.cc:237] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.550288 1804 consensus_queue.cc:237] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.563465 1377 catalog_manager.cc:5495] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:19.564841 1377 catalog_manager.cc:5495] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 2 to 3, leader changed from 2a8943ab8566448487437f6561e08900 (127.0.204.131) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:19.565951 1377 catalog_manager.cc:5495] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:19.588963 1799 raft_consensus.cc:979] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.589501 1795 raft_consensus.cc:1067] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.590979 1480 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47020
I20260927 19:37:19.591481 1480 raft_consensus.cc:482] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:19.592187 1480 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:37:19.597788 1480 raft_consensus.cc:504] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.599351 1480 leader_election.cc:288] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 4 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:19.600107 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:19.600322 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:19.600598 1554 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:37:19.600867 1629 raft_consensus.cc:3032] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:37:19.601119 1629 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:19.601805 1629 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.602876 1629 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:37:19.605060 1554 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 4.
I20260927 19:37:19.606009 1444 leader_election.cc:302] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:19.606660 1804 raft_consensus.cc:2781] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:37:19.607069 1804 raft_consensus.cc:686] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:19.607873 1804 consensus_queue.cc:237] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.608803 1629 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 4.
I20260927 19:37:19.615772 1377 catalog_manager.cc:5495] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 3 to 4, leader changed from 2a8943ab8566448487437f6561e08900 (127.0.204.131) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:19.655256 1793 raft_consensus.cc:979] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: : Instructing follower 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.655717 1800 raft_consensus.cc:1067] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Signalling peer 2703f921034946859b3f26666c525028 to start an election
I20260927 19:37:19.657305 1480 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd"
dest_uuid: "2703f921034946859b3f26666c525028"
from {username='slave'} at 127.0.0.1:47030
I20260927 19:37:19.657821 1480 raft_consensus.cc:482] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:19.658115 1480 raft_consensus.cc:3037] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.662978 1480 raft_consensus.cc:504] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.664415 1480 leader_election.cc:288] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Requested vote from peers 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:19.665392 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:19.665535 1629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ca2fe4531954caba51132a547d82bfd" candidate_uuid: "2703f921034946859b3f26666c525028" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:19.665993 1629 raft_consensus.cc:3037] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.665993 1554 raft_consensus.cc:3032] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:19.666463 1554 raft_consensus.cc:729] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:19.667174 1554 consensus_queue.cc:260] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.668336 1554 raft_consensus.cc:3037] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:19.672441 1629 raft_consensus.cc:2454] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:19.673553 1446 leader_election.cc:302] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2703f921034946859b3f26666c525028, 2a8943ab8566448487437f6561e08900; no voters:
I20260927 19:37:19.674265 1804 raft_consensus.cc:2781] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:19.674268 1554 raft_consensus.cc:2454] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2703f921034946859b3f26666c525028 in term 2.
I20260927 19:37:19.675768 1804 raft_consensus.cc:686] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Leader. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:19.676687 1804 consensus_queue.cc:237] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:19.684361 1377 catalog_manager.cc:5495] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 reported cstate change: term changed from 1 to 2, leader changed from 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130) to 2703f921034946859b3f26666c525028 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "2703f921034946859b3f26666c525028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:19.935542 1629 raft_consensus.cc:1261] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:19.936651 1804 consensus_queue.cc:1035] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:19.942802 1554 raft_consensus.cc:1261] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:19.944087 1801 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:19.949707 1555 raft_consensus.cc:1261] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:19.951836 1803 consensus_queue.cc:1035] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:19.961514 1629 raft_consensus.cc:1261] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:19.963685 1803 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:19.983865 1554 raft_consensus.cc:1261] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:19.985087 1803 consensus_queue.cc:1035] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:19.995483 1629 raft_consensus.cc:1261] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:19.997097 1804 consensus_queue.cc:1035] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:20.016037 1554 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:37:20.017134 1804 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:20.027766 1629 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:37:20.029580 1796 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:20.132216 1554 raft_consensus.cc:1261] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:20.133442 1796 consensus_queue.cc:1035] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:20.143126 1629 raft_consensus.cc:1261] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Refusing update from remote peer 2703f921034946859b3f26666c525028: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:37:20.144369 1796 consensus_queue.cc:1035] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:22.402453 818 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260927 19:37:22.403487 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:22.412299 1480 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0dc94069046c49f2b373529add98b664"
dest_uuid: "2703f921034946859b3f26666c525028"
mode: GRACEFUL
new_leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
from {username='slave'} at 127.0.0.1:47044
I20260927 19:37:22.412958 1480 raft_consensus.cc:595] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Received request to transfer leadership to TS 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:22.415391 1480 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "69e273a93f5b488c98762a6950a85486"
dest_uuid: "2703f921034946859b3f26666c525028"
mode: GRACEFUL
new_leader_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:47044
I20260927 19:37:22.415946 1480 raft_consensus.cc:595] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 LEADER]: Received request to transfer leadership to TS 2a8943ab8566448487437f6561e08900
I20260927 19:37:22.418037 1480 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2703f921034946859b3f26666c525028"
mode: GRACEFUL
new_leader_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:47044
I20260927 19:37:22.418432 1480 raft_consensus.cc:595] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 LEADER]: Received request to transfer leadership to TS 2a8943ab8566448487437f6561e08900
I20260927 19:37:22.420419 1480 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c"
dest_uuid: "2703f921034946859b3f26666c525028"
mode: GRACEFUL
new_leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
from {username='slave'} at 127.0.0.1:47044
I20260927 19:37:22.420821 1480 raft_consensus.cc:595] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Received request to transfer leadership to TS 6e54647c35ee47b48d8c7a4bf445ba14
I20260927 19:37:22.421873 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260927 19:37:22.422410 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:22.507121 1796 raft_consensus.cc:979] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: : Instructing follower 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:22.507884 1829 raft_consensus.cc:1067] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 LEADER]: Signalling peer 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:22.509397 1628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3"
dest_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:44330
I20260927 19:37:22.510064 1628 raft_consensus.cc:482] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:22.510459 1628 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:37:22.517295 1628 raft_consensus.cc:504] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.519858 1628 leader_election.cc:288] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 5 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:22.520768 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:22.521085 1555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "960861c418c145508e7dc07cc2e571c3" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:22.521706 1480 raft_consensus.cc:3032] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:37:22.521736 1555 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:37:22.522099 1480 raft_consensus.cc:729] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:22.522979 1480 consensus_queue.cc:260] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.524030 1480 raft_consensus.cc:3037] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:37:22.527748 1555 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 5.
I20260927 19:37:22.529103 1593 leader_election.cc:302] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:22.530122 1795 raft_consensus.cc:2781] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:37:22.530429 1480 raft_consensus.cc:2454] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 5.
I20260927 19:37:22.531600 1795 raft_consensus.cc:686] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 5 LEADER]: Becoming Leader. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:22.532476 1795 consensus_queue.cc:237] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.540282 1377 catalog_manager.cc:5495] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 reported cstate change: term changed from 4 to 5, leader changed from 2703f921034946859b3f26666c525028 (127.0.204.129) to 2a8943ab8566448487437f6561e08900 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "2a8943ab8566448487437f6561e08900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:22.566953 1796 raft_consensus.cc:979] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: : Instructing follower 6e54647c35ee47b48d8c7a4bf445ba14 to start an election
I20260927 19:37:22.567399 1829 raft_consensus.cc:1067] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Signalling peer 6e54647c35ee47b48d8c7a4bf445ba14 to start an election
I20260927 19:37:22.568687 1555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0dc94069046c49f2b373529add98b664"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
from {username='slave'} at 127.0.0.1:44396
I20260927 19:37:22.569217 1555 raft_consensus.cc:482] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:22.569521 1555 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:22.574657 1555 raft_consensus.cc:504] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.576633 1555 leader_election.cc:288] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 3 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:22.577546 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:22.577682 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc94069046c49f2b373529add98b664" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:22.578064 1480 raft_consensus.cc:3032] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:37:22.578220 1628 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:22.578393 1480 raft_consensus.cc:729] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:22.579155 1480 consensus_queue.cc:260] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.580032 1480 raft_consensus.cc:3037] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:22.583205 1628 raft_consensus.cc:2454] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 3.
I20260927 19:37:22.584339 1480 raft_consensus.cc:2454] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 3.
I20260927 19:37:22.584403 1521 leader_election.cc:302] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:22.585482 1793 raft_consensus.cc:2781] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:37:22.586514 1793 raft_consensus.cc:686] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:22.587184 1793 consensus_queue.cc:237] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.594210 1377 catalog_manager.cc:5495] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 2 to 3, leader changed from 2703f921034946859b3f26666c525028 (127.0.204.129) to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:22.706422 1796 raft_consensus.cc:979] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: : Instructing follower 6e54647c35ee47b48d8c7a4bf445ba14 to start an election
I20260927 19:37:22.707034 1829 raft_consensus.cc:1067] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Signalling peer 6e54647c35ee47b48d8c7a4bf445ba14 to start an election
I20260927 19:37:22.708492 1554 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c"
dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
from {username='slave'} at 127.0.0.1:44396
I20260927 19:37:22.708993 1554 raft_consensus.cc:482] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:22.709213 1554 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:22.713850 1554 raft_consensus.cc:504] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.715301 1554 leader_election.cc:288] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 3 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 2a8943ab8566448487437f6561e08900 (127.0.204.131:36223)
I20260927 19:37:22.716221 1628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2a8943ab8566448487437f6561e08900"
I20260927 19:37:22.716197 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e893d17f33d9498cb4d31f52016ab66c" candidate_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:22.716918 1480 raft_consensus.cc:3032] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:37:22.716887 1628 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:22.717338 1480 raft_consensus.cc:729] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:22.718127 1480 consensus_queue.cc:260] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.719394 1480 raft_consensus.cc:3037] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:22.723071 1796 raft_consensus.cc:979] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: : Instructing follower 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:22.723582 1829 raft_consensus.cc:1067] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 LEADER]: Signalling peer 2a8943ab8566448487437f6561e08900 to start an election
I20260927 19:37:22.723966 1628 raft_consensus.cc:2454] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 3.
I20260927 19:37:22.725112 1521 leader_election.cc:302] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:22.725821 1480 raft_consensus.cc:2454] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e54647c35ee47b48d8c7a4bf445ba14 in term 3.
I20260927 19:37:22.725997 1793 raft_consensus.cc:2781] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:37:22.726083 1628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "69e273a93f5b488c98762a6950a85486"
dest_uuid: "2a8943ab8566448487437f6561e08900"
from {username='slave'} at 127.0.0.1:44330
I20260927 19:37:22.726707 1793 raft_consensus.cc:686] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 LEADER]: Becoming Leader. State: Replica: 6e54647c35ee47b48d8c7a4bf445ba14, State: Running, Role: LEADER
I20260927 19:37:22.726760 1628 raft_consensus.cc:482] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:37:22.727243 1628 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:37:22.727682 1793 consensus_queue.cc:237] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.734952 1628 raft_consensus.cc:504] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.735531 1374 catalog_manager.cc:5495] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 reported cstate change: term changed from 2 to 3, leader changed from 2703f921034946859b3f26666c525028 (127.0.204.129) to 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:22.736788 1628 leader_election.cc:288] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 4 election: Requested vote from peers 2703f921034946859b3f26666c525028 (127.0.204.129:45337), 6e54647c35ee47b48d8c7a4bf445ba14 (127.0.204.130:46457)
I20260927 19:37:22.737454 1480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2703f921034946859b3f26666c525028"
I20260927 19:37:22.737993 1480 raft_consensus.cc:3032] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:37:22.738291 1480 raft_consensus.cc:729] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2703f921034946859b3f26666c525028, State: Running, Role: LEADER
I20260927 19:37:22.738282 1554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e273a93f5b488c98762a6950a85486" candidate_uuid: "2a8943ab8566448487437f6561e08900" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6e54647c35ee47b48d8c7a4bf445ba14"
I20260927 19:37:22.738961 1554 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:37:22.739171 1480 consensus_queue.cc:260] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.740432 1480 raft_consensus.cc:3037] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:37:22.745138 1554 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 4.
I20260927 19:37:22.746222 1593 leader_election.cc:302] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8943ab8566448487437f6561e08900, 6e54647c35ee47b48d8c7a4bf445ba14; no voters:
I20260927 19:37:22.746348 1480 raft_consensus.cc:2454] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8943ab8566448487437f6561e08900 in term 4.
I20260927 19:37:22.747004 1795 raft_consensus.cc:2781] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:37:22.747481 1795 raft_consensus.cc:686] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 4 LEADER]: Becoming Leader. State: Replica: 2a8943ab8566448487437f6561e08900, State: Running, Role: LEADER
I20260927 19:37:22.748260 1795 consensus_queue.cc:237] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } }
I20260927 19:37:22.755650 1374 catalog_manager.cc:5495] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 reported cstate change: term changed from 3 to 4, leader changed from 2703f921034946859b3f26666c525028 (127.0.204.129) to 2a8943ab8566448487437f6561e08900 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "2a8943ab8566448487437f6561e08900" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:22.966619 1480 raft_consensus.cc:1261] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Refusing update from remote peer 6e54647c35ee47b48d8c7a4bf445ba14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:22.967839 1793 consensus_queue.cc:1035] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:22.980861 1628 raft_consensus.cc:1261] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Refusing update from remote peer 6e54647c35ee47b48d8c7a4bf445ba14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:22.983142 1842 consensus_queue.cc:1035] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:22.988302 1554 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 5 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:37:22.989822 1795 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:37:23.000561 1480 raft_consensus.cc:1261] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 5 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:37:23.001900 1795 consensus_queue.cc:1035] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:37:23.144796 1628 raft_consensus.cc:1261] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Refusing update from remote peer 6e54647c35ee47b48d8c7a4bf445ba14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:23.146103 1793 consensus_queue.cc:1035] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8943ab8566448487437f6561e08900" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:23.151171 1480 raft_consensus.cc:1261] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:37:23.152465 1847 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:23.157563 1479 raft_consensus.cc:1261] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Refusing update from remote peer 6e54647c35ee47b48d8c7a4bf445ba14: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:23.159113 1793 consensus_queue.cc:1035] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2703f921034946859b3f26666c525028" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:23.162144 1554 raft_consensus.cc:1261] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Refusing update from remote peer 2a8943ab8566448487437f6561e08900: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:37:23.163484 1847 consensus_queue.cc:1035] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e54647c35ee47b48d8c7a4bf445ba14" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:23.423339 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:23.429272 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:23.429731 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:24.430554 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:24.435974 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:24.436460 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:25.437191 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:25.442476 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:25.442924 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:26.443737 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:26.448797 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:26.449246 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:27.450054 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:27.455066 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:27.455498 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:28.456444 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:28.462067 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:28.462531 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:29.463352 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:29.468688 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:29.469120 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:30.470196 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:30.475592 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:30.476028 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:31.476920 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:37:31.482196 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260927 19:37:31.482816 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:37:32.515416 818 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260927 19:37:32.516120 818 tablet_server.cc:178] TabletServer@127.0.204.129:0 shutting down...
I20260927 19:37:32.548101 818 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 19:37:32.548839 818 tablet_replica.cc:330] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.549624 818 raft_consensus.cc:2229] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.550572 818 raft_consensus.cc:2258] T cc5273de00ed4b70b5ee53cabc03c689 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.552729 818 tablet_replica.cc:330] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.553277 818 raft_consensus.cc:2229] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.554037 818 raft_consensus.cc:2258] T 5165cd333794441ab37c5bd95004b9a1 P 2703f921034946859b3f26666c525028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.556023 818 tablet_replica.cc:330] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.556632 818 raft_consensus.cc:2229] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.557126 818 raft_consensus.cc:2258] T ba423fb866274e838a9389da2ed20677 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.559144 818 tablet_replica.cc:330] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.559619 818 raft_consensus.cc:2229] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.560086 818 raft_consensus.cc:2258] T e893d17f33d9498cb4d31f52016ab66c P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.562000 818 tablet_replica.cc:330] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.562441 818 raft_consensus.cc:2229] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.563329 818 raft_consensus.cc:2258] T a89edbac45f14ad399aafe8ea1932989 P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.565152 818 tablet_replica.cc:330] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.565809 818 raft_consensus.cc:2229] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.566897 818 raft_consensus.cc:2258] T 82cd60fd586944ea9949e6cbd910624c P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.568917 818 tablet_replica.cc:330] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.569464 818 raft_consensus.cc:2229] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.570192 818 raft_consensus.cc:2258] T 69e273a93f5b488c98762a6950a85486 P 2703f921034946859b3f26666c525028 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.571967 818 tablet_replica.cc:330] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.572510 818 raft_consensus.cc:2229] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.573098 818 raft_consensus.cc:2258] T 0dc94069046c49f2b373529add98b664 P 2703f921034946859b3f26666c525028 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.575070 818 tablet_replica.cc:330] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.575762 818 raft_consensus.cc:2229] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.576745 818 raft_consensus.cc:2258] T 4ca2fe4531954caba51132a547d82bfd P 2703f921034946859b3f26666c525028 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.578595 818 tablet_replica.cc:330] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028: stopping tablet replica
I20260927 19:37:32.579005 818 raft_consensus.cc:2229] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.579416 818 raft_consensus.cc:2258] T 960861c418c145508e7dc07cc2e571c3 P 2703f921034946859b3f26666c525028 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.613659 818 tablet_server.cc:194] TabletServer@127.0.204.129:0 shutdown complete.
I20260927 19:37:32.634047 818 tablet_server.cc:178] TabletServer@127.0.204.130:0 shutting down...
I20260927 19:37:32.666949 818 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 19:37:32.667726 818 tablet_replica.cc:330] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.668318 818 raft_consensus.cc:2229] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.668804 818 raft_consensus.cc:2258] T ba423fb866274e838a9389da2ed20677 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.670658 818 tablet_replica.cc:330] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.671202 818 raft_consensus.cc:2229] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.672042 818 raft_consensus.cc:2258] T 5165cd333794441ab37c5bd95004b9a1 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.673914 818 tablet_replica.cc:330] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.674412 818 raft_consensus.cc:2229] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.674820 818 raft_consensus.cc:2258] T 82cd60fd586944ea9949e6cbd910624c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.676566 818 tablet_replica.cc:330] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.677040 818 raft_consensus.cc:2229] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.677561 818 raft_consensus.cc:2258] T 69e273a93f5b488c98762a6950a85486 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.679169 818 tablet_replica.cc:330] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.679626 818 raft_consensus.cc:2229] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.680042 818 raft_consensus.cc:2258] T cc5273de00ed4b70b5ee53cabc03c689 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.681646 818 tablet_replica.cc:330] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.682145 818 raft_consensus.cc:2229] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.682569 818 raft_consensus.cc:2258] T 4ca2fe4531954caba51132a547d82bfd P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.684263 818 tablet_replica.cc:330] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.684741 818 raft_consensus.cc:2229] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.685150 818 raft_consensus.cc:2258] T 960861c418c145508e7dc07cc2e571c3 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.686861 818 tablet_replica.cc:330] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.687377 818 raft_consensus.cc:2229] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.688266 818 raft_consensus.cc:2258] T 0dc94069046c49f2b373529add98b664 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.690039 818 tablet_replica.cc:330] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.690497 818 raft_consensus.cc:2229] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.690903 818 raft_consensus.cc:2258] T a89edbac45f14ad399aafe8ea1932989 P 6e54647c35ee47b48d8c7a4bf445ba14 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.692637 818 tablet_replica.cc:330] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14: stopping tablet replica
I20260927 19:37:32.693102 818 raft_consensus.cc:2229] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.693866 818 raft_consensus.cc:2258] T e893d17f33d9498cb4d31f52016ab66c P 6e54647c35ee47b48d8c7a4bf445ba14 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260927 19:37:32.706172 1592 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.204.129:45337: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20260927 19:37:32.711877 1592 consensus_peers.cc:487] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 -> Peer 2703f921034946859b3f26666c525028 (127.0.204.129:45337): Couldn't send request to peer 2703f921034946859b3f26666c525028. Status: Network error: Client connection negotiation failed: client connection to 127.0.204.129:45337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:32.751410 818 tablet_server.cc:194] TabletServer@127.0.204.130:0 shutdown complete.
I20260927 19:37:32.771701 818 tablet_server.cc:178] TabletServer@127.0.204.131:0 shutting down...
I20260927 19:37:32.803961 818 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260927 19:37:32.804720 818 tablet_replica.cc:330] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.805335 818 raft_consensus.cc:2229] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.805898 818 raft_consensus.cc:2258] T 82cd60fd586944ea9949e6cbd910624c P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.807821 818 tablet_replica.cc:330] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.808364 818 raft_consensus.cc:2229] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 2 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.809188 818 raft_consensus.cc:2258] T ba423fb866274e838a9389da2ed20677 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.811151 818 tablet_replica.cc:330] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.811678 818 raft_consensus.cc:2229] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.812065 818 raft_consensus.cc:2258] T 5165cd333794441ab37c5bd95004b9a1 P 2a8943ab8566448487437f6561e08900 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.813897 818 tablet_replica.cc:330] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.814432 818 raft_consensus.cc:2229] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 4 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.815236 818 raft_consensus.cc:2258] T 69e273a93f5b488c98762a6950a85486 P 2a8943ab8566448487437f6561e08900 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.817129 818 tablet_replica.cc:330] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.817984 818 raft_consensus.cc:2229] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.818498 818 raft_consensus.cc:2258] T e893d17f33d9498cb4d31f52016ab66c P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.820279 818 tablet_replica.cc:330] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.820787 818 raft_consensus.cc:2229] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.821223 818 raft_consensus.cc:2258] T a89edbac45f14ad399aafe8ea1932989 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.822968 818 tablet_replica.cc:330] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.823460 818 raft_consensus.cc:2229] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.823889 818 raft_consensus.cc:2258] T 0dc94069046c49f2b373529add98b664 P 2a8943ab8566448487437f6561e08900 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.825593 818 tablet_replica.cc:330] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.826076 818 raft_consensus.cc:2229] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.826491 818 raft_consensus.cc:2258] T 4ca2fe4531954caba51132a547d82bfd P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.828347 818 tablet_replica.cc:330] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.828826 818 raft_consensus.cc:2229] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 5 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.829695 818 raft_consensus.cc:2258] T 960861c418c145508e7dc07cc2e571c3 P 2a8943ab8566448487437f6561e08900 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.831449 818 tablet_replica.cc:330] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900: stopping tablet replica
I20260927 19:37:32.831895 818 raft_consensus.cc:2229] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:32.832319 818 raft_consensus.cc:2258] T cc5273de00ed4b70b5ee53cabc03c689 P 2a8943ab8566448487437f6561e08900 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.862926 818 tablet_server.cc:194] TabletServer@127.0.204.131:0 shutdown complete.
I20260927 19:37:32.882923 818 master.cc:537] Master@127.0.204.190:36435 shutting down...
I20260927 19:37:32.908056 818 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 1 LEADER]: Raft consensus shutting down.
I20260927 19:37:32.908584 818 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:32.908880 818 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7fd0855939b84ca1a4a229994b18a745: stopping tablet replica
I20260927 19:37:32.929934 818 master.cc:552] Master@127.0.204.190:36435 shutdown complete.
2703f921034946859b3f26666c525028 5
2a8943ab8566448487437f6561e08900 1
6e54647c35ee47b48d8c7a4bf445ba14 4
2703f921034946859b3f26666c525028 3
2a8943ab8566448487437f6561e08900 3
6e54647c35ee47b48d8c7a4bf445ba14 4
2703f921034946859b3f26666c525028 8
2a8943ab8566448487437f6561e08900 1
6e54647c35ee47b48d8c7a4bf445ba14 1
2703f921034946859b3f26666c525028 4
2a8943ab8566448487437f6561e08900 3
6e54647c35ee47b48d8c7a4bf445ba14 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29686 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260927 19:37:32.970408 818 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.204.190:41277
I20260927 19:37:32.971799 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:32.978750 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:32.978750 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:32.982951 818 server_base.cc:1034] running on GCE node
W20260927 19:37:32.983376 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:32.984308 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:32.984494 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:32.984601 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537852984593 us; error 0 us; skew 500 ppm
I20260927 19:37:32.985038 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:32.992234 818 webserver.cc:460] Webserver started at http://127.0.204.190:32805/ using document root <none> and password file <none>
I20260927 19:37:32.992704 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:32.992851 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:32.993079 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:32.994287 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/master-0-root/instance:
uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0"
format_stamp: "Formatted at 2026-09-27 19:37:32 on dist-test-slave-zwk0"
I20260927 19:37:32.998807 818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 19:37:33.002521 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.003392 818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 19:37:33.003778 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/master-0-root
uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0"
format_stamp: "Formatted at 2026-09-27 19:37:32 on dist-test-slave-zwk0"
I20260927 19:37:33.004127 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:33.022416 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:33.023677 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:33.065392 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.190:41277
I20260927 19:37:33.065586 1927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.190:41277 every 8 connection(s)
I20260927 19:37:33.070335 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.081046 1928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Bootstrap starting.
I20260927 19:37:33.085342 1928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:33.090230 1928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: No bootstrap required, opened a new log
I20260927 19:37:33.092338 1928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" member_type: VOTER }
I20260927 19:37:33.092831 1928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:33.093086 1928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0ff9812dcb4ed79a1d0e8f390f6cb0, State: Initialized, Role: FOLLOWER
I20260927 19:37:33.093628 1928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" member_type: VOTER }
I20260927 19:37:33.094086 1928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 19:37:33.094314 1928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 19:37:33.094626 1928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:33.098817 1928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" member_type: VOTER }
I20260927 19:37:33.099380 1928 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0ff9812dcb4ed79a1d0e8f390f6cb0; no voters:
I20260927 19:37:33.100538 1928 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 19:37:33.100996 1931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:33.102712 1931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 0a0ff9812dcb4ed79a1d0e8f390f6cb0, State: Running, Role: LEADER
I20260927 19:37:33.103539 1928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 19:37:33.103546 1931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" member_type: VOTER }
I20260927 19:37:33.109422 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0ff9812dcb4ed79a1d0e8f390f6cb0. Latest consensus state: current_term: 1 leader_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" member_type: VOTER } }
I20260927 19:37:33.109390 1932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0ff9812dcb4ed79a1d0e8f390f6cb0" member_type: VOTER } }
I20260927 19:37:33.110226 1932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [sys.catalog]: This master's current role is: LEADER
I20260927 19:37:33.110214 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0 [sys.catalog]: This master's current role is: LEADER
I20260927 19:37:33.116168 1939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 19:37:33.123740 1939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 19:37:33.125013 818 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260927 19:37:33.133735 1939 catalog_manager.cc:1347] Generated new cluster ID: 4b6ba4d7ef8c4da0a8c4cf137c262110
I20260927 19:37:33.134147 1939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 19:37:33.149993 1939 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 19:37:33.151777 1939 catalog_manager.cc:1504] Loading token signing keys...
I20260927 19:37:33.165230 1939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Generated new TSK 0
I20260927 19:37:33.165956 1939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 19:37:33.191918 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:33.198000 1950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:33.199301 1951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:33.202414 1953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:33.202440 818 server_base.cc:1034] running on GCE node
I20260927 19:37:33.203658 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:33.203876 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:33.204057 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537853204041 us; error 0 us; skew 500 ppm
I20260927 19:37:33.204576 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:33.207432 818 webserver.cc:460] Webserver started at http://127.0.204.129:43515/ using document root <none> and password file <none>
I20260927 19:37:33.207967 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:33.208204 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:33.208504 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:33.209789 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/instance:
uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
format_stamp: "Formatted at 2026-09-27 19:37:33 on dist-test-slave-zwk0"
I20260927 19:37:33.214934 818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 19:37:33.218816 1958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.219655 818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 19:37:33.219969 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root
uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
format_stamp: "Formatted at 2026-09-27 19:37:33 on dist-test-slave-zwk0"
I20260927 19:37:33.220259 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:33.241436 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:33.242689 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:33.244135 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:33.246464 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:33.246685 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.246901 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:33.247076 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.288398 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.129:39941
I20260927 19:37:33.288501 2020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.129:39941 every 8 connection(s)
I20260927 19:37:33.293310 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:33.300262 2025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:33.302119 2026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:33.304169 2028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:33.304513 818 server_base.cc:1034] running on GCE node
I20260927 19:37:33.306442 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:33.306716 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:33.306880 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537853306867 us; error 0 us; skew 500 ppm
I20260927 19:37:33.307400 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:33.309437 2021 heartbeater.cc:347] Connected to a master server at 127.0.204.190:41277
I20260927 19:37:33.309846 2021 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:33.310459 818 webserver.cc:460] Webserver started at http://127.0.204.130:37667/ using document root <none> and password file <none>
I20260927 19:37:33.310755 2021 heartbeater.cc:511] Master 127.0.204.190:41277 requested a full tablet report, sending...
I20260927 19:37:33.311084 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:33.311336 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:33.311636 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:33.312793 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/instance:
uuid: "4cc2df048f424e08803a390946004df0"
format_stamp: "Formatted at 2026-09-27 19:37:33 on dist-test-slave-zwk0"
I20260927 19:37:33.313027 1893 ts_manager.cc:194] Registered new tserver with Master: 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:33.315012 1893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46382
I20260927 19:37:33.317981 818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 19:37:33.322847 2033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.323922 818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 19:37:33.324306 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root
uuid: "4cc2df048f424e08803a390946004df0"
format_stamp: "Formatted at 2026-09-27 19:37:33 on dist-test-slave-zwk0"
I20260927 19:37:33.324697 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:33.348083 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:33.349255 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:33.350778 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:33.353029 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:33.353308 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.353610 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:33.353820 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.394699 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.130:45515
I20260927 19:37:33.394789 2095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.130:45515 every 8 connection(s)
I20260927 19:37:33.399504 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:33.407431 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:33.408847 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:33.411988 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:33.415761 818 server_base.cc:1034] running on GCE node
I20260927 19:37:33.417043 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:33.417304 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:33.417448 2096 heartbeater.cc:347] Connected to a master server at 127.0.204.190:41277
I20260927 19:37:33.417558 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537853417541 us; error 0 us; skew 500 ppm
I20260927 19:37:33.417902 2096 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:33.418249 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:33.418608 2096 heartbeater.cc:511] Master 127.0.204.190:41277 requested a full tablet report, sending...
I20260927 19:37:33.420948 1893 ts_manager.cc:194] Registered new tserver with Master: 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:33.421030 818 webserver.cc:460] Webserver started at http://127.0.204.131:46441/ using document root <none> and password file <none>
I20260927 19:37:33.421730 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:33.421980 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:33.422277 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:33.422745 1893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46386
I20260927 19:37:33.423472 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/instance:
uuid: "69f09866d0a1477585f05b2f82ff4c67"
format_stamp: "Formatted at 2026-09-27 19:37:33 on dist-test-slave-zwk0"
I20260927 19:37:33.428225 818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 19:37:33.431690 2107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.432513 818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 19:37:33.432837 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root
uuid: "69f09866d0a1477585f05b2f82ff4c67"
format_stamp: "Formatted at 2026-09-27 19:37:33 on dist-test-slave-zwk0"
I20260927 19:37:33.433135 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:33.453841 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:33.455030 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:33.456409 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:33.458647 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:33.458820 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.459123 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:33.459265 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:33.499243 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.131:45045
I20260927 19:37:33.499328 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.131:45045 every 8 connection(s)
I20260927 19:37:33.515547 2170 heartbeater.cc:347] Connected to a master server at 127.0.204.190:41277
I20260927 19:37:33.515956 2170 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:33.516773 2170 heartbeater.cc:511] Master 127.0.204.190:41277 requested a full tablet report, sending...
I20260927 19:37:33.518723 1893 ts_manager.cc:194] Registered new tserver with Master: 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:33.519253 818 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017011834s
I20260927 19:37:33.520419 1893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46400
I20260927 19:37:33.524489 818 test_util.cc:268] Using random seed: 347560435
I20260927 19:37:33.551180 1893 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46408:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 19:37:33.557435 1893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 19:37:33.753983 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 303e2824ebc54b4c9bce740f15e0105d (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260927 19:37:33.754514 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 96289a4006ee4b44977df34d8b39798a (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260927 19:37:33.755635 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e2824ebc54b4c9bce740f15e0105d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.757551 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 85f45e6282f545f19d655ea989e9e123 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260927 19:37:33.757640 1983 tablet_service.cc:1467] Processing CreateTablet for tablet acb3a2a0b9b047249cd5a7535a946fe9 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260927 19:37:33.758308 1982 tablet_service.cc:1467] Processing CreateTablet for tablet c0017d74ca734f828f8096d7d474a209 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260927 19:37:33.759243 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f45e6282f545f19d655ea989e9e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.759879 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0017d74ca734f828f8096d7d474a209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.761164 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289a4006ee4b44977df34d8b39798a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.761407 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4fa5fe3da244bb9e845557555810b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260927 19:37:33.761870 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3a2a0b9b047249cd5a7535a946fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.765848 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fa5fe3da244bb9e845557555810b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.771631 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 303e2824ebc54b4c9bce740f15e0105d (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260927 19:37:33.773212 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e2824ebc54b4c9bce740f15e0105d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.773722 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 85f45e6282f545f19d655ea989e9e123 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260927 19:37:33.775170 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f45e6282f545f19d655ea989e9e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.778716 2132 tablet_service.cc:1467] Processing CreateTablet for tablet acb3a2a0b9b047249cd5a7535a946fe9 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260927 19:37:33.780258 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3a2a0b9b047249cd5a7535a946fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.782677 2131 tablet_service.cc:1467] Processing CreateTablet for tablet c0017d74ca734f828f8096d7d474a209 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260927 19:37:33.784144 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0017d74ca734f828f8096d7d474a209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.789470 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 303e2824ebc54b4c9bce740f15e0105d (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260927 19:37:33.791033 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e2824ebc54b4c9bce740f15e0105d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.794687 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 96289a4006ee4b44977df34d8b39798a (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260927 19:37:33.796131 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289a4006ee4b44977df34d8b39798a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.798663 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 85f45e6282f545f19d655ea989e9e123 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260927 19:37:33.801055 2058 tablet_service.cc:1467] Processing CreateTablet for tablet acb3a2a0b9b047249cd5a7535a946fe9 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260927 19:37:33.803439 2057 tablet_service.cc:1467] Processing CreateTablet for tablet c0017d74ca734f828f8096d7d474a209 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260927 19:37:33.805857 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4fa5fe3da244bb9e845557555810b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260927 19:37:33.806581 2055 tablet_service.cc:1467] Processing CreateTablet for tablet ac63cb1bcdd141bb9de9fcb5d0f102af (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260927 19:37:33.771811 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 96289a4006ee4b44977df34d8b39798a (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260927 19:37:33.808300 2130 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4fa5fe3da244bb9e845557555810b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260927 19:37:33.810039 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac63cb1bcdd141bb9de9fcb5d0f102af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.810632 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fa5fe3da244bb9e845557555810b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.811605 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96289a4006ee4b44977df34d8b39798a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.812849 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 32bc1524f46f4197b6ac8d29ea44ab77 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260927 19:37:33.814388 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bc1524f46f4197b6ac8d29ea44ab77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.814261 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 32bc1524f46f4197b6ac8d29ea44ab77 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260927 19:37:33.816833 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 32bc1524f46f4197b6ac8d29ea44ab77 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260927 19:37:33.818321 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bc1524f46f4197b6ac8d29ea44ab77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.818955 2053 tablet_service.cc:1467] Processing CreateTablet for tablet d811e58acb4b4a3aa93ab155fd5ab1c2 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260927 19:37:33.820446 2127 tablet_service.cc:1467] Processing CreateTablet for tablet d811e58acb4b4a3aa93ab155fd5ab1c2 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260927 19:37:33.825984 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 4667efc6973d414fbff5011f6ceaa9b1 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260927 19:37:33.827450 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4667efc6973d414fbff5011f6ceaa9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.828078 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 4667efc6973d414fbff5011f6ceaa9b1 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260927 19:37:33.830834 1979 tablet_service.cc:1467] Processing CreateTablet for tablet d811e58acb4b4a3aa93ab155fd5ab1c2 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260927 19:37:33.831161 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 4667efc6973d414fbff5011f6ceaa9b1 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260927 19:37:33.832315 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d811e58acb4b4a3aa93ab155fd5ab1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.832795 1977 tablet_service.cc:1467] Processing CreateTablet for tablet c2df5ab506c34b2c974fdefdf324e734 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260927 19:37:33.801056 1985 tablet_service.cc:1467] Processing CreateTablet for tablet ac63cb1bcdd141bb9de9fcb5d0f102af (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260927 19:37:33.842062 2129 tablet_service.cc:1467] Processing CreateTablet for tablet ac63cb1bcdd141bb9de9fcb5d0f102af (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260927 19:37:33.845714 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fa5fe3da244bb9e845557555810b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.847929 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d811e58acb4b4a3aa93ab155fd5ab1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.848162 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2df5ab506c34b2c974fdefdf324e734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.849277 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32bc1524f46f4197b6ac8d29ea44ab77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.850630 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4667efc6973d414fbff5011f6ceaa9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.853789 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0017d74ca734f828f8096d7d474a209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.857764 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f45e6282f545f19d655ea989e9e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.861754 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3a2a0b9b047249cd5a7535a946fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.865762 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac63cb1bcdd141bb9de9fcb5d0f102af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.873732 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d811e58acb4b4a3aa93ab155fd5ab1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.877810 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac63cb1bcdd141bb9de9fcb5d0f102af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.879858 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4667efc6973d414fbff5011f6ceaa9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.906993 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 876ffc23594245239e2688f5992774cf (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260927 19:37:33.908475 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876ffc23594245239e2688f5992774cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.926848 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 29707548f67846318643f37690b25103 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260927 19:37:33.928189 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29707548f67846318643f37690b25103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.950228 2060 tablet_service.cc:1467] Processing CreateTablet for tablet c2df5ab506c34b2c974fdefdf324e734 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260927 19:37:33.950698 2052 tablet_service.cc:1467] Processing CreateTablet for tablet 29707548f67846318643f37690b25103 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260927 19:37:33.951646 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2df5ab506c34b2c974fdefdf324e734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.952387 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29707548f67846318643f37690b25103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.953505 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 876ffc23594245239e2688f5992774cf (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260927 19:37:33.973340 1982 tablet_service.cc:1467] Processing CreateTablet for tablet ac5251fc915042bab4ce94b1c17deb87 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260927 19:37:33.981014 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5251fc915042bab4ce94b1c17deb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.985164 2131 tablet_service.cc:1467] Processing CreateTablet for tablet c2df5ab506c34b2c974fdefdf324e734 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260927 19:37:33.987038 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2df5ab506c34b2c974fdefdf324e734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.974251 2185 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:33.973604 2132 tablet_service.cc:1467] Processing CreateTablet for tablet 29707548f67846318643f37690b25103 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260927 19:37:33.991150 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0e603e501b4a8e9e2fe63d3784d744 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260927 19:37:33.992802 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29707548f67846318643f37690b25103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.976866 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876ffc23594245239e2688f5992774cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.976323 2054 tablet_service.cc:1467] Processing CreateTablet for tablet ac5251fc915042bab4ce94b1c17deb87 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260927 19:37:34.022327 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0e603e501b4a8e9e2fe63d3784d744 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260927 19:37:34.023787 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0e603e501b4a8e9e2fe63d3784d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.023428 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 3d15dfddc2cb4edfaef7625c7eba492c (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260927 19:37:34.026261 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d15dfddc2cb4edfaef7625c7eba492c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.026837 2133 tablet_service.cc:1467] Processing CreateTablet for tablet ac5251fc915042bab4ce94b1c17deb87 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260927 19:37:34.028309 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5251fc915042bab4ce94b1c17deb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.031641 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0e603e501b4a8e9e2fe63d3784d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:33.966890 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 876ffc23594245239e2688f5992774cf (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260927 19:37:34.033963 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876ffc23594245239e2688f5992774cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.034457 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5251fc915042bab4ce94b1c17deb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.037842 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 3d15dfddc2cb4edfaef7625c7eba492c (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260927 19:37:34.037828 2186 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:34.048719 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d15dfddc2cb4edfaef7625c7eba492c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.051445 2053 tablet_service.cc:1467] Processing CreateTablet for tablet f61659c573e142aa9c3fbb673f3e15f0 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260927 19:37:34.052958 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61659c573e142aa9c3fbb673f3e15f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.038688 2187 tablet_bootstrap.cc:492] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:34.057744 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 3d15dfddc2cb4edfaef7625c7eba492c (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260927 19:37:34.082396 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d15dfddc2cb4edfaef7625c7eba492c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.058284 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 1b0e603e501b4a8e9e2fe63d3784d744 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260927 19:37:34.059250 2185 tablet_bootstrap.cc:654] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:34.084703 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b0e603e501b4a8e9e2fe63d3784d744. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.086294 1983 tablet_service.cc:1467] Processing CreateTablet for tablet f61659c573e142aa9c3fbb673f3e15f0 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260927 19:37:34.089417 1978 tablet_service.cc:1467] Processing CreateTablet for tablet d9ad83fb0df9449f9081439026d2964f (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260927 19:37:34.078454 2057 tablet_service.cc:1467] Processing CreateTablet for tablet d9ad83fb0df9449f9081439026d2964f (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260927 19:37:34.105510 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ad83fb0df9449f9081439026d2964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.055099 2186 tablet_bootstrap.cc:654] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:34.118309 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:34.119464 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 794b9dbf053c445393588ff152553db5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260927 19:37:34.120011 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 794b9dbf053c445393588ff152553db5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260927 19:37:34.121448 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794b9dbf053c445393588ff152553db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.123762 2127 tablet_service.cc:1467] Processing CreateTablet for tablet f61659c573e142aa9c3fbb673f3e15f0 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260927 19:37:34.130832 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0cc9e9ed38443aa7b5b0a629befdcd (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260927 19:37:34.132313 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0cc9e9ed38443aa7b5b0a629befdcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.134295 2130 tablet_service.cc:1467] Processing CreateTablet for tablet d9ad83fb0df9449f9081439026d2964f (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260927 19:37:34.137634 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61659c573e142aa9c3fbb673f3e15f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.137638 2058 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcc7a171b3e443d8f28fd70f8742470 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260927 19:37:34.154729 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ad83fb0df9449f9081439026d2964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.157984 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc7a171b3e443d8f28fd70f8742470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.143373 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0cc9e9ed38443aa7b5b0a629befdcd (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260927 19:37:34.165282 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0cc9e9ed38443aa7b5b0a629befdcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.182386 2052 tablet_service.cc:1467] Processing CreateTablet for tablet ba36610baf2046bc9b0bd15df9129f16 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260927 19:37:34.183841 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba36610baf2046bc9b0bd15df9129f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.191322 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 63522eab31e04c8191338bb22acb0297 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260927 19:37:34.192883 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63522eab31e04c8191338bb22acb0297. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:34.201714 1879 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 69480.5us
W20260927 19:37:34.204752 1880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 60035.9us
I20260927 19:37:34.205713 2185 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:34.206250 2185 ts_tablet_manager.cc:1397] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.232s user 0.006s sys 0.012s
I20260927 19:37:34.209339 2185 raft_consensus.cc:348] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.210319 2185 raft_consensus.cc:374] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:34.210660 2185 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:34.211553 2185 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.125243 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f61659c573e142aa9c3fbb673f3e15f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.217886 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794b9dbf053c445393588ff152553db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.217186 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ad83fb0df9449f9081439026d2964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.213312 2187 tablet_bootstrap.cc:654] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:34.261123 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0cc9e9ed38443aa7b5b0a629befdcd (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260927 19:37:34.263356 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0cc9e9ed38443aa7b5b0a629befdcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.268678 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcc7a171b3e443d8f28fd70f8742470 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260927 19:37:34.270571 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc7a171b3e443d8f28fd70f8742470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.183712 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcc7a171b3e443d8f28fd70f8742470 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260927 19:37:34.283840 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc7a171b3e443d8f28fd70f8742470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.291293 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec895e575154c488c4fb6666ae75ce3 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260927 19:37:34.218775 1982 tablet_service.cc:1467] Processing CreateTablet for tablet 63522eab31e04c8191338bb22acb0297 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260927 19:37:34.222811 1986 tablet_service.cc:1467] Processing CreateTablet for tablet ba36610baf2046bc9b0bd15df9129f16 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260927 19:37:34.219574 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 901577b40c874966936247e62860f154 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
W20260927 19:37:34.288959 1880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 53005.6us
I20260927 19:37:34.319783 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec895e575154c488c4fb6666ae75ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.324859 2021 heartbeater.cc:503] Master 127.0.204.190:41277 was elected leader, sending a full tablet report...
I20260927 19:37:34.327138 2185 ts_tablet_manager.cc:1428] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.121s user 0.048s sys 0.043s
I20260927 19:37:34.328150 2185 tablet_bootstrap.cc:492] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:34.330564 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec895e575154c488c4fb6666ae75ce3 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260927 19:37:34.295476 1977 tablet_service.cc:1467] Processing CreateTablet for tablet 901577b40c874966936247e62860f154 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260927 19:37:34.332413 2187 tablet_bootstrap.cc:492] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:34.333004 2187 ts_tablet_manager.cc:1397] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.295s user 0.008s sys 0.038s
I20260927 19:37:34.333791 2132 tablet_service.cc:1467] Processing CreateTablet for tablet 63522eab31e04c8191338bb22acb0297 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260927 19:37:34.334355 2126 tablet_service.cc:1467] Processing CreateTablet for tablet ba36610baf2046bc9b0bd15df9129f16 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260927 19:37:34.335186 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63522eab31e04c8191338bb22acb0297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.336131 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba36610baf2046bc9b0bd15df9129f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.320467 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 794b9dbf053c445393588ff152553db5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260927 19:37:34.338119 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 794b9dbf053c445393588ff152553db5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:34.323328 1879 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 88775.4us
I20260927 19:37:34.348256 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec895e575154c488c4fb6666ae75ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.353278 2186 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:34.353513 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 901577b40c874966936247e62860f154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.353982 2186 ts_tablet_manager.cc:1397] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.317s user 0.031s sys 0.022s
I20260927 19:37:34.357508 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63522eab31e04c8191338bb22acb0297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.362135 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 901577b40c874966936247e62860f154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.371554 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba36610baf2046bc9b0bd15df9129f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.404337 2187 raft_consensus.cc:348] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.406386 2187 raft_consensus.cc:374] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:34.407174 2187 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
W20260927 19:37:34.409509 1880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 78703.6us
I20260927 19:37:34.412907 2185 tablet_bootstrap.cc:654] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:34.417637 1879 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 78284.7us
I20260927 19:37:34.420395 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 901577b40c874966936247e62860f154 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260927 19:37:34.421784 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 901577b40c874966936247e62860f154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.424252 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 3134bec8f5e84b64bcb7237586476295 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260927 19:37:34.425056 2058 tablet_service.cc:1467] Processing CreateTablet for tablet 069da74585d54798b7b8d625c46ae785 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260927 19:37:34.425527 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3134bec8f5e84b64bcb7237586476295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.426376 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069da74585d54798b7b8d625c46ae785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.428982 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 45a655bc810741ac9d95bf0a2cb4b759 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260927 19:37:34.429541 2186 raft_consensus.cc:348] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:34.430305 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a655bc810741ac9d95bf0a2cb4b759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.430563 2186 raft_consensus.cc:374] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:34.430920 2186 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:34.432732 2186 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:34.434394 2056 tablet_service.cc:1467] Processing CreateTablet for tablet aabcfbc05a724a19857afe1f8211259f (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260927 19:37:34.435348 2096 heartbeater.cc:503] Master 127.0.204.190:41277 was elected leader, sending a full tablet report...
I20260927 19:37:34.466322 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabcfbc05a724a19857afe1f8211259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.435400 2134 tablet_service.cc:1467] Processing CreateTablet for tablet c6886530f9f741c2acaff96949f65f19 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260927 19:37:34.468617 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6886530f9f741c2acaff96949f65f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.435866 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 069da74585d54798b7b8d625c46ae785 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260927 19:37:34.438967 2052 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed94cb8c6c4a0badc3ff336e7b2968 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260927 19:37:34.439147 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 6ec895e575154c488c4fb6666ae75ce3 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260927 19:37:34.436439 1980 tablet_service.cc:1467] Processing CreateTablet for tablet c6886530f9f741c2acaff96949f65f19 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260927 19:37:34.475865 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6886530f9f741c2acaff96949f65f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.475867 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed94cb8c6c4a0badc3ff336e7b2968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.489624 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 3134bec8f5e84b64bcb7237586476295 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260927 19:37:34.409003 2187 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.490617 2054 tablet_service.cc:1467] Processing CreateTablet for tablet a4214caa28574754b61aad89471c77b9 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260927 19:37:34.502272 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 069da74585d54798b7b8d625c46ae785 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260927 19:37:34.458446 2053 tablet_service.cc:1467] Processing CreateTablet for tablet c6886530f9f741c2acaff96949f65f19 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260927 19:37:34.506623 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 45a655bc810741ac9d95bf0a2cb4b759 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260927 19:37:34.508324 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3134bec8f5e84b64bcb7237586476295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.514938 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069da74585d54798b7b8d625c46ae785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.522280 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4214caa28574754b61aad89471c77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.522812 2058 tablet_service.cc:1467] Processing CreateTablet for tablet ff14f359ca724a70853a0490b2f2c680 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260927 19:37:34.524384 2170 heartbeater.cc:503] Master 127.0.204.190:41277 was elected leader, sending a full tablet report...
I20260927 19:37:34.524559 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 3725aebc9b4c41a793e21109302696ec (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260927 19:37:34.527395 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 39d534ed586444279970bb03de60c919 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260927 19:37:34.529383 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff14f359ca724a70853a0490b2f2c680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.530386 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3725aebc9b4c41a793e21109302696ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.533216 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069da74585d54798b7b8d625c46ae785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.533632 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec895e575154c488c4fb6666ae75ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.533749 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a655bc810741ac9d95bf0a2cb4b759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.539403 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6886530f9f741c2acaff96949f65f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.542225 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 3134bec8f5e84b64bcb7237586476295 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260927 19:37:34.543529 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3134bec8f5e84b64bcb7237586476295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.549431 1979 tablet_service.cc:1467] Processing CreateTablet for tablet aabcfbc05a724a19857afe1f8211259f (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260927 19:37:34.550781 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabcfbc05a724a19857afe1f8211259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.552120 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d534ed586444279970bb03de60c919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.571810 2193 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:34.572554 2193 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:34.570668 2190 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:34.579928 2190 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.576642 2187 ts_tablet_manager.cc:1428] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.243s user 0.013s sys 0.031s
I20260927 19:37:34.582252 2190 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:34.589679 2187 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:34.598877 2193 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:34.602115 2194 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:34.602811 2194 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.622514 2185 tablet_bootstrap.cc:492] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:34.623042 2185 ts_tablet_manager.cc:1397] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.295s user 0.025s sys 0.037s
I20260927 19:37:34.627534 2134 tablet_service.cc:1467] Processing CreateTablet for tablet aabcfbc05a724a19857afe1f8211259f (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260927 19:37:34.629925 1982 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed94cb8c6c4a0badc3ff336e7b2968 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260927 19:37:34.633917 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 45a655bc810741ac9d95bf0a2cb4b759 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260927 19:37:34.659775 1984 tablet_service.cc:1467] Processing CreateTablet for tablet ff14f359ca724a70853a0490b2f2c680 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260927 19:37:34.643755 2133 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed94cb8c6c4a0badc3ff336e7b2968 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260927 19:37:34.650960 2194 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:34.661206 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff14f359ca724a70853a0490b2f2c680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.672272 2127 tablet_service.cc:1467] Processing CreateTablet for tablet ff14f359ca724a70853a0490b2f2c680 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260927 19:37:34.681934 2055 tablet_service.cc:1467] Processing CreateTablet for tablet d0f797e07d634f6abe29556a11863581 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260927 19:37:34.684211 2056 tablet_service.cc:1467] Processing CreateTablet for tablet b2f6360a6b784953bc89cd72712a5e7d (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260927 19:37:34.685264 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed94cb8c6c4a0badc3ff336e7b2968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.691552 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45a655bc810741ac9d95bf0a2cb4b759. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.699364 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed94cb8c6c4a0badc3ff336e7b2968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.652904 1980 tablet_service.cc:1467] Processing CreateTablet for tablet a4214caa28574754b61aad89471c77b9 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260927 19:37:34.704777 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff14f359ca724a70853a0490b2f2c680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.714301 2185 raft_consensus.cc:348] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.715113 2185 raft_consensus.cc:374] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:34.715395 2185 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:34.716099 2185 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.718525 2185 ts_tablet_manager.cc:1428] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.095s user 0.006s sys 0.002s
I20260927 19:37:34.726217 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f797e07d634f6abe29556a11863581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.744314 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 7f97d36ebc9149818eb4749186e672e7 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260927 19:37:34.745868 2052 tablet_service.cc:1467] Processing CreateTablet for tablet f6ecc556dac3458bb7b5ffda19a96158 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260927 19:37:34.747176 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecc556dac3458bb7b5ffda19a96158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.756227 2130 tablet_service.cc:1467] Processing CreateTablet for tablet a4214caa28574754b61aad89471c77b9 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260927 19:37:34.757700 2186 ts_tablet_manager.cc:1428] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.403s user 0.016s sys 0.002s
I20260927 19:37:34.758380 2054 tablet_service.cc:1467] Processing CreateTablet for tablet f30da008e35141e99c8237c84d8e66b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260927 19:37:34.759326 2185 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:34.757730 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 39d534ed586444279970bb03de60c919 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260927 19:37:34.767287 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d534ed586444279970bb03de60c919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.770032 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f97d36ebc9149818eb4749186e672e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.777725 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f6360a6b784953bc89cd72712a5e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.778270 1985 tablet_service.cc:1467] Processing CreateTablet for tablet b2f6360a6b784953bc89cd72712a5e7d (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260927 19:37:34.779645 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f6360a6b784953bc89cd72712a5e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.756101 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 3725aebc9b4c41a793e21109302696ec (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260927 19:37:34.790940 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3725aebc9b4c41a793e21109302696ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.792331 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4214caa28574754b61aad89471c77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.780864 1983 tablet_service.cc:1467] Processing CreateTablet for tablet d0f797e07d634f6abe29556a11863581 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260927 19:37:34.793900 2186 tablet_bootstrap.cc:492] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:34.795070 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f797e07d634f6abe29556a11863581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.796563 2185 tablet_bootstrap.cc:654] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:34.798430 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30da008e35141e99c8237c84d8e66b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.799940 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabcfbc05a724a19857afe1f8211259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.806175 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4214caa28574754b61aad89471c77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.828504 1977 tablet_service.cc:1467] Processing CreateTablet for tablet f30da008e35141e99c8237c84d8e66b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260927 19:37:34.809773 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 33febda9075249dc9de325af8884cce5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260927 19:37:34.836992 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33febda9075249dc9de325af8884cce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.839322 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 7f97d36ebc9149818eb4749186e672e7 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260927 19:37:34.840569 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f97d36ebc9149818eb4749186e672e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.843770 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 39d534ed586444279970bb03de60c919 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260927 19:37:34.843938 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 3725aebc9b4c41a793e21109302696ec (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260927 19:37:34.845253 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d534ed586444279970bb03de60c919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.845711 2132 tablet_service.cc:1467] Processing CreateTablet for tablet b2f6360a6b784953bc89cd72712a5e7d (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260927 19:37:34.846938 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f6360a6b784953bc89cd72712a5e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.849916 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3725aebc9b4c41a793e21109302696ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.850565 1978 tablet_service.cc:1467] Processing CreateTablet for tablet f6ecc556dac3458bb7b5ffda19a96158 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260927 19:37:34.850749 2128 tablet_service.cc:1467] Processing CreateTablet for tablet d0f797e07d634f6abe29556a11863581 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260927 19:37:34.854336 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0f797e07d634f6abe29556a11863581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.859163 2190 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:34.859664 2190 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.861670 2190 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:34.864686 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:34.865566 2144 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:34.866976 1962 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:34.867111 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:34.868674 1962 leader_election.cc:341] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:34.869345 2190 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:34.869742 2190 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:34.870077 2190 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:34.871625 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:34.878511 2059 tablet_service.cc:1467] Processing CreateTablet for tablet c802eef7b2904ec984925e3999cff361 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260927 19:37:34.879865 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c802eef7b2904ec984925e3999cff361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.881613 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30da008e35141e99c8237c84d8e66b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.906165 2057 tablet_service.cc:1467] Processing CreateTablet for tablet efcb4aec6b174f11b08d4483d9f63866 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260927 19:37:34.908434 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecc556dac3458bb7b5ffda19a96158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.908236 2054 tablet_service.cc:1467] Processing CreateTablet for tablet da05114b545f4ec9a25e86d9c059fea3 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260927 19:37:34.876926 2190 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:34.913106 2187 tablet_bootstrap.cc:700] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Time spent opening tablet: real 0.104s user 0.003s sys 0.000s
I20260927 19:37:34.914016 2187 tablet_bootstrap.cc:654] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:34.915401 2061 tablet_service.cc:1467] Processing CreateTablet for tablet db442f51d759423ba3bde8876ae46aa0 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260927 19:37:34.911247 2186 tablet_bootstrap.cc:654] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:34.930902 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 7f97d36ebc9149818eb4749186e672e7 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260927 19:37:34.932340 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f97d36ebc9149818eb4749186e672e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.933730 1996 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:37:34.933967 2135 tablet_service.cc:1467] Processing CreateTablet for tablet efcb4aec6b174f11b08d4483d9f63866 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260927 19:37:34.936262 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcb4aec6b174f11b08d4483d9f63866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.942718 2129 tablet_service.cc:1467] Processing CreateTablet for tablet f30da008e35141e99c8237c84d8e66b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260927 19:37:34.944167 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f30da008e35141e99c8237c84d8e66b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.947180 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 33febda9075249dc9de325af8884cce5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260927 19:37:34.951927 2133 tablet_service.cc:1467] Processing CreateTablet for tablet f6ecc556dac3458bb7b5ffda19a96158 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260927 19:37:34.953385 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecc556dac3458bb7b5ffda19a96158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.955158 2185 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:34.970902 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33febda9075249dc9de325af8884cce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.971943 2187 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:34.970960 2185 ts_tablet_manager.cc:1397] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.212s user 0.013s sys 0.001s
I20260927 19:37:34.971521 1982 tablet_service.cc:1467] Processing CreateTablet for tablet 2034953084704d748cf5bd91b19f1a46 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260927 19:37:34.974751 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2034953084704d748cf5bd91b19f1a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:34.972470 2187 ts_tablet_manager.cc:1397] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.386s user 0.014s sys 0.005s
I20260927 19:37:35.006131 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:35.008551 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:35.009940 1961 leader_election.cc:341] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.010401 1961 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.014304 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:35.014984 2145 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.023361 2190 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.044610 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db442f51d759423ba3bde8876ae46aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.046716 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0ad2bfdf76425782a9548c96da6ca7 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260927 19:37:35.048102 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ad2bfdf76425782a9548c96da6ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.048591 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:35.046821 2145 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:35.046123 2185 raft_consensus.cc:348] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.055012 2058 tablet_service.cc:1467] Processing CreateTablet for tablet 2034953084704d748cf5bd91b19f1a46 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260927 19:37:35.055502 2190 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.055253 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 56cc0a374f434fc193a1c4fc0edcf8c5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260927 19:37:35.057303 2186 tablet_bootstrap.cc:492] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.057911 2186 ts_tablet_manager.cc:1397] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.264s user 0.014s sys 0.004s
I20260927 19:37:35.064286 1962 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:35.062844 2186 raft_consensus.cc:348] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.065843 2186 raft_consensus.cc:374] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.066736 2186 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.057461 2052 tablet_service.cc:1467] Processing CreateTablet for tablet f19d4799cf134d51b543a2952f564adc (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260927 19:37:35.069303 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcb4aec6b174f11b08d4483d9f63866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.072196 1985 tablet_service.cc:1467] Processing CreateTablet for tablet c802eef7b2904ec984925e3999cff361 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260927 19:37:35.074800 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 83f97251c21d4e979bce8419c208369c (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260927 19:37:35.065200 2190 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:35.082455 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.050690 2185 raft_consensus.cc:374] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.088033 2058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2034953084704d748cf5bd91b19f1a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.112716 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0ad2bfdf76425782a9548c96da6ca7 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260927 19:37:35.113677 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c802eef7b2904ec984925e3999cff361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.114064 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ad2bfdf76425782a9548c96da6ca7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:35.113940 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 103641us
W20260927 19:37:35.097880 2037 leader_election.cc:341] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.087297 2132 tablet_service.cc:1467] Processing CreateTablet for tablet 33febda9075249dc9de325af8884cce5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260927 19:37:35.089986 2185 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
W20260927 19:37:35.128541 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:35.129024 2052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19d4799cf134d51b543a2952f564adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.137084 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cc0a374f434fc193a1c4fc0edcf8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.141693 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05114b545f4ec9a25e86d9c059fea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.087297 2134 tablet_service.cc:1467] Processing CreateTablet for tablet db442f51d759423ba3bde8876ae46aa0 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260927 19:37:35.144712 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33febda9075249dc9de325af8884cce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.134403 2059 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe684e9563d491b9d4b3b99204450c8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260927 19:37:35.146129 2059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe684e9563d491b9d4b3b99204450c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:35.133586 2037 leader_election.cc:341] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.146214 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 1c40cea829004940809b7bbd894de6b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260927 19:37:35.146548 2037 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.132596 1977 tablet_service.cc:1467] Processing CreateTablet for tablet da05114b545f4ec9a25e86d9c059fea3 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260927 19:37:35.147689 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c40cea829004940809b7bbd894de6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.148283 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05114b545f4ec9a25e86d9c059fea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.153689 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f97251c21d4e979bce8419c208369c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.084306 2187 raft_consensus.cc:348] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.157475 2187 raft_consensus.cc:374] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.157871 2187 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.158648 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 8814088a5337420ab29ac2b4b4e0f314 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260927 19:37:35.160231 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8814088a5337420ab29ac2b4b4e0f314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.118232 2186 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.160935 2193 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.161389 2193 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.089435 2130 tablet_service.cc:1467] Processing CreateTablet for tablet c802eef7b2904ec984925e3999cff361 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260927 19:37:35.100711 1979 tablet_service.cc:1467] Processing CreateTablet for tablet db442f51d759423ba3bde8876ae46aa0 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260927 19:37:35.163718 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db442f51d759423ba3bde8876ae46aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.118396 2190 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:35.158720 2187 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.173941 2190 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.174319 2193 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.174403 1980 tablet_service.cc:1467] Processing CreateTablet for tablet efcb4aec6b174f11b08d4483d9f63866 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260927 19:37:35.175314 2193 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.175981 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcb4aec6b174f11b08d4483d9f63866. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:35.176556 1961 leader_election.cc:341] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.177938 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db442f51d759423ba3bde8876ae46aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.103113 2128 tablet_service.cc:1467] Processing CreateTablet for tablet da05114b545f4ec9a25e86d9c059fea3 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260927 19:37:35.189749 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c802eef7b2904ec984925e3999cff361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.190377 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 56cc0a374f434fc193a1c4fc0edcf8c5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260927 19:37:35.191584 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cc0a374f434fc193a1c4fc0edcf8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.191401 2060 tablet_service.cc:1467] Processing CreateTablet for tablet 9e017d73273c4a83bcfdb4c95caa32df (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260927 19:37:35.191957 2135 tablet_service.cc:1467] Processing CreateTablet for tablet f19d4799cf134d51b543a2952f564adc (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260927 19:37:35.192963 2060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e017d73273c4a83bcfdb4c95caa32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.134604 2185 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.177953 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 56cc0a374f434fc193a1c4fc0edcf8c5 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260927 19:37:35.195904 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cc0a374f434fc193a1c4fc0edcf8c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.201570 2126 tablet_service.cc:1467] Processing CreateTablet for tablet 2034953084704d748cf5bd91b19f1a46 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260927 19:37:35.202456 1982 tablet_service.cc:1467] Processing CreateTablet for tablet f19d4799cf134d51b543a2952f564adc (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260927 19:37:35.203804 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19d4799cf134d51b543a2952f564adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.214556 2211 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.193158 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19d4799cf134d51b543a2952f564adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.215075 2211 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:35.226066 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 92408.1us
I20260927 19:37:35.227352 2185 ts_tablet_manager.cc:1428] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.254s user 0.008s sys 0.004s
I20260927 19:37:35.228396 2185 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.230780 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:35.230808 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:35.232203 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:35.231937 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0ad2bfdf76425782a9548c96da6ca7 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260927 19:37:35.233395 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ad2bfdf76425782a9548c96da6ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.244336 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.249387 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:35.250317 1961 leader_election.cc:341] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:35.251331 1961 leader_election.cc:341] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.254249 2055 tablet_service.cc:1467] Processing CreateTablet for tablet 81be6adca85345a3b402f5d474ec3e5a (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260927 19:37:35.256718 2186 ts_tablet_manager.cc:1428] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.198s user 0.007s sys 0.009s
I20260927 19:37:35.258474 2186 tablet_bootstrap.cc:492] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
W20260927 19:37:35.261722 2037 leader_election.cc:341] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:35.262795 2110 leader_election.cc:341] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.263245 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:35.279399 1983 tablet_service.cc:1467] Processing CreateTablet for tablet 83f97251c21d4e979bce8419c208369c (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260927 19:37:35.281172 1983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f97251c21d4e979bce8419c208369c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.283617 1890 catalog_manager.cc:5495] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:35.284387 1979 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe684e9563d491b9d4b3b99204450c8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
W20260927 19:37:35.285300 2037 leader_election.cc:341] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.293448 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2034953084704d748cf5bd91b19f1a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.304046 2133 tablet_service.cc:1467] Processing CreateTablet for tablet 83f97251c21d4e979bce8419c208369c (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260927 19:37:35.301842 2055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81be6adca85345a3b402f5d474ec3e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.301714 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.309629 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:35.310478 2212 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.311205 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05114b545f4ec9a25e86d9c059fea3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:35.311507 1962 leader_election.cc:341] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.311931 1962 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.312767 2190 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.313858 2133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f97251c21d4e979bce8419c208369c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.300787 2211 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.315831 2129 tablet_service.cc:1467] Processing CreateTablet for tablet 4fe684e9563d491b9d4b3b99204450c8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260927 19:37:35.317157 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe684e9563d491b9d4b3b99204450c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 19:37:35.329522 2022 tablet.cc:2369] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 19:37:35.329049 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 66420.7us
I20260927 19:37:35.335548 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 1c40cea829004940809b7bbd894de6b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260927 19:37:35.336974 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c40cea829004940809b7bbd894de6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.341068 2134 tablet_service.cc:1467] Processing CreateTablet for tablet 8814088a5337420ab29ac2b4b4e0f314 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260927 19:37:35.341868 2185 tablet_bootstrap.cc:654] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.342429 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8814088a5337420ab29ac2b4b4e0f314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.348588 2127 tablet_service.cc:1467] Processing CreateTablet for tablet 9e017d73273c4a83bcfdb4c95caa32df (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260927 19:37:35.349920 2127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e017d73273c4a83bcfdb4c95caa32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.358459 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260927 19:37:35.359776 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.362921 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 3acf4c2d78fb41ad974812ebf41d0478 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260927 19:37:35.364207 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acf4c2d78fb41ad974812ebf41d0478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.370599 2056 tablet_service.cc:1467] Processing CreateTablet for tablet 0505ba7a2573409cbd518d77ccbcd201 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260927 19:37:35.371976 1979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe684e9563d491b9d4b3b99204450c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.373823 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 8814088a5337420ab29ac2b4b4e0f314 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260927 19:37:35.375120 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8814088a5337420ab29ac2b4b4e0f314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.319789 1984 tablet_service.cc:1467] Processing CreateTablet for tablet 1c40cea829004940809b7bbd894de6b8 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260927 19:37:35.386797 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c40cea829004940809b7bbd894de6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.403501 2056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0505ba7a2573409cbd518d77ccbcd201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.405644 2132 tablet_service.cc:1467] Processing CreateTablet for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260927 19:37:35.407209 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.416442 1977 tablet_service.cc:1467] Processing CreateTablet for tablet 9e017d73273c4a83bcfdb4c95caa32df (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260927 19:37:35.423162 2216 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.423699 2216 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.426079 2061 tablet_service.cc:1467] Processing CreateTablet for tablet 228a1cf257434cbe9e5616dbd94e1864 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260927 19:37:35.427476 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228a1cf257434cbe9e5616dbd94e1864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.429160 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 81be6adca85345a3b402f5d474ec3e5a (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260927 19:37:35.437989 2187 ts_tablet_manager.cc:1428] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.461s user 0.021s sys 0.031s
I20260927 19:37:35.439463 2187 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.442163 2216 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.442695 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.443377 1996 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:35.445847 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:35.422672 2186 tablet_bootstrap.cc:654] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.453558 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81be6adca85345a3b402f5d474ec3e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.462872 2217 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.457504 2216 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:35.398790 2130 tablet_service.cc:1467] Processing CreateTablet for tablet 81be6adca85345a3b402f5d474ec3e5a (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260927 19:37:35.480309 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 3acf4c2d78fb41ad974812ebf41d0478 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260927 19:37:35.480994 2130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81be6adca85345a3b402f5d474ec3e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.480358 2131 tablet_service.cc:1467] Processing CreateTablet for tablet 228a1cf257434cbe9e5616dbd94e1864 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260927 19:37:35.484814 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acf4c2d78fb41ad974812ebf41d0478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.489552 1981 tablet_service.cc:1467] Processing CreateTablet for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260927 19:37:35.469456 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:35.492044 2110 leader_election.cc:341] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.496464 1980 tablet_service.cc:1467] Processing CreateTablet for tablet 228a1cf257434cbe9e5616dbd94e1864 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260927 19:37:35.494702 1986 tablet_service.cc:1467] Processing CreateTablet for tablet 0505ba7a2573409cbd518d77ccbcd201 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260927 19:37:35.498062 2135 tablet_service.cc:1467] Processing CreateTablet for tablet 0505ba7a2573409cbd518d77ccbcd201 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260927 19:37:35.499306 2135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0505ba7a2573409cbd518d77ccbcd201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.468909 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e017d73273c4a83bcfdb4c95caa32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.510092 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.468343 2216 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:35.526106 2131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228a1cf257434cbe9e5616dbd94e1864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.527213 1986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0505ba7a2573409cbd518d77ccbcd201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.525684 2216 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.531935 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228a1cf257434cbe9e5616dbd94e1864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.533231 2185 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:35.533895 2185 ts_tablet_manager.cc:1397] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.306s user 0.010s sys 0.026s
I20260927 19:37:35.539075 1985 tablet_service.cc:1467] Processing CreateTablet for tablet 3acf4c2d78fb41ad974812ebf41d0478 (DEFAULT_TABLE table=test-workload [id=613fa53ad5754ab495032b9d9d2d4862]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260927 19:37:35.540446 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acf4c2d78fb41ad974812ebf41d0478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:35.537237 2187 tablet_bootstrap.cc:654] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.536896 2185 raft_consensus.cc:348] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.556648 2186 tablet_bootstrap.cc:492] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.557142 2186 ts_tablet_manager.cc:1397] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.300s user 0.014s sys 0.019s
I20260927 19:37:35.551057 2185 raft_consensus.cc:374] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.571623 2185 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.574680 2185 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.559757 2186 raft_consensus.cc:348] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.575800 2186 raft_consensus.cc:374] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.576139 2186 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.576993 2186 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.579205 2185 ts_tablet_manager.cc:1428] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260927 19:37:35.580281 2185 tablet_bootstrap.cc:492] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.583987 2186 ts_tablet_manager.cc:1428] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260927 19:37:35.585093 2186 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:35.592816 2216 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.598398 2216 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.599176 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:35.599696 1996 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.594867 2185 tablet_bootstrap.cc:654] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.605731 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:35.606901 1996 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
W20260927 19:37:35.607246 2110 leader_election.cc:341] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.608086 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:37:35.609388 2216 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:35.613081 2216 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:35.613987 2187 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:35.614645 2187 ts_tablet_manager.cc:1397] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.175s user 0.021s sys 0.000s
I20260927 19:37:35.614040 2216 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.598651 2186 tablet_bootstrap.cc:654] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.616276 2185 tablet_bootstrap.cc:492] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:35.616973 2185 ts_tablet_manager.cc:1397] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.000s
I20260927 19:37:35.620262 2185 raft_consensus.cc:348] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.620651 2187 raft_consensus.cc:348] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.621167 2185 raft_consensus.cc:374] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.621445 2187 raft_consensus.cc:374] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.621532 2185 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.621840 2187 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.622535 2185 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.622642 2187 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.625667 2187 ts_tablet_manager.cc:1428] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260927 19:37:35.628355 2185 ts_tablet_manager.cc:1428] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260927 19:37:35.628589 2187 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.631839 1892 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:35.635517 2186 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.635608 2185 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.636184 2186 ts_tablet_manager.cc:1397] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.003s
I20260927 19:37:35.639148 2186 raft_consensus.cc:348] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.639979 2186 raft_consensus.cc:374] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.640303 2186 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.642580 2069 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.645490 2185 tablet_bootstrap.cc:654] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.645149 2186 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.652117 2211 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:35.652964 2187 tablet_bootstrap.cc:654] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.652015 2186 ts_tablet_manager.cc:1428] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260927 19:37:35.655885 2186 tablet_bootstrap.cc:492] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:35.664204 2186 tablet_bootstrap.cc:654] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.667538 2187 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:35.668072 2187 ts_tablet_manager.cc:1397] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.005s
I20260927 19:37:35.674152 2185 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:35.674747 2185 ts_tablet_manager.cc:1397] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.012s
I20260927 19:37:35.678224 2186 tablet_bootstrap.cc:492] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.678771 2186 ts_tablet_manager.cc:1397] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.004s
I20260927 19:37:35.679411 2185 raft_consensus.cc:348] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.679998 2213 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:35.680470 2185 raft_consensus.cc:374] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.680891 2185 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.681743 2186 raft_consensus.cc:348] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.682605 2186 raft_consensus.cc:374] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.681941 2185 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.682952 2186 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.683832 2186 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.671033 2187 raft_consensus.cc:348] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.688223 2187 raft_consensus.cc:374] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.688606 2187 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.689457 2187 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.693215 2213 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.693810 2213 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.696441 2186 ts_tablet_manager.cc:1428] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260927 19:37:35.697002 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:35.698549 1962 leader_election.cc:341] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.701555 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:35.702265 2069 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:35.702560 2186 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:35.703591 1961 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.704777 2211 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:35.705197 2211 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:35.705624 2211 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.708765 2213 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.710031 2186 tablet_bootstrap.cc:654] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.710285 2185 ts_tablet_manager.cc:1428] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
I20260927 19:37:35.717725 2187 ts_tablet_manager.cc:1428] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20260927 19:37:35.719125 2187 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.719799 2185 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.732728 2186 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.733242 2187 tablet_bootstrap.cc:654] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.733263 2186 ts_tablet_manager.cc:1397] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.012s
I20260927 19:37:35.736666 2185 tablet_bootstrap.cc:654] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.737106 2186 raft_consensus.cc:348] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.738190 2186 raft_consensus.cc:374] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.738567 2186 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.740626 2211 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.744448 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:37:35.746387 1962 leader_election.cc:341] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.748997 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:35.748982 2186 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.749786 2070 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.750260 2211 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.752614 2186 ts_tablet_manager.cc:1428] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260927 19:37:35.753690 2186 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:35.757287 2070 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:35.758591 1961 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.759406 2211 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:35.759895 2186 tablet_bootstrap.cc:654] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.759876 2211 raft_consensus.cc:686] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:35.760800 2211 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.761842 2187 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:35.762293 2187 ts_tablet_manager.cc:1397] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.012s
I20260927 19:37:35.765038 2185 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:35.765583 2185 ts_tablet_manager.cc:1397] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.015s
I20260927 19:37:35.765290 2187 raft_consensus.cc:348] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.766168 2187 raft_consensus.cc:374] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.766479 2187 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.767153 2187 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.768496 2185 raft_consensus.cc:348] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.769229 2185 raft_consensus.cc:374] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.769531 2185 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.770290 2187 ts_tablet_manager.cc:1428] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 19:37:35.770213 2185 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.771543 2187 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.772886 2185 ts_tablet_manager.cc:1428] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:35.773901 2185 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.774525 2213 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.775020 2213 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.777071 2213 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.778360 2187 tablet_bootstrap.cc:654] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.779031 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:35.779800 2185 tablet_bootstrap.cc:654] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.780510 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:35.781224 2070 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
W20260927 19:37:35.782454 1962 leader_election.cc:341] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.783299 1961 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.784405 2211 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:35.784760 2211 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:35.785076 2211 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.791483 1892 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:35.796787 2213 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.797422 2213 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.801776 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:35.804426 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:35.804659 1962 leader_election.cc:341] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.805200 2213 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.805145 2070 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:35.796289 2186 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.806142 2186 ts_tablet_manager.cc:1397] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.000s
I20260927 19:37:35.806535 1961 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.807641 2213 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:35.807976 2213 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:35.808271 2213 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.808887 2186 raft_consensus.cc:348] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.811600 2186 raft_consensus.cc:374] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.811908 2186 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.812954 2211 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.815137 2213 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.815793 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:35.816334 2070 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.819519 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:35.820127 2069 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.824103 2070 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:35.825313 1961 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:35.825071 2186 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.826120 2190 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:35.829088 2186 ts_tablet_manager.cc:1428] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260927 19:37:35.830932 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:35.832100 2211 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:37:35.832401 1962 leader_election.cc:341] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.835803 2186 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:35.837471 2190 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:35.836331 2187 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:35.838245 2187 ts_tablet_manager.cc:1397] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.067s user 0.025s sys 0.007s
I20260927 19:37:35.838742 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:35.838341 2190 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.839779 2185 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
W20260927 19:37:35.840026 1962 leader_election.cc:341] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.840270 2185 ts_tablet_manager.cc:1397] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.067s user 0.013s sys 0.049s
I20260927 19:37:35.840622 2213 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.833179 2069 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:35.843200 1961 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.843240 2187 raft_consensus.cc:348] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.844108 2187 raft_consensus.cc:374] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.844394 2187 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.844571 2213 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:35.845198 2187 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.847445 2186 tablet_bootstrap.cc:654] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.847435 2187 ts_tablet_manager.cc:1428] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260927 19:37:35.849198 2187 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.845223 2213 raft_consensus.cc:686] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:35.850819 2185 raft_consensus.cc:348] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.851557 2185 raft_consensus.cc:374] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.851137 2213 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.851840 2185 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.852694 2185 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.855799 2185 ts_tablet_manager.cc:1428] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20260927 19:37:35.856846 2187 tablet_bootstrap.cc:654] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.858487 2185 tablet_bootstrap.cc:492] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.870283 2187 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:35.870504 2185 tablet_bootstrap.cc:654] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.870796 2187 ts_tablet_manager.cc:1397] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.002s
I20260927 19:37:35.872197 1893 catalog_manager.cc:5495] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:35.874101 2187 raft_consensus.cc:348] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.874825 2187 raft_consensus.cc:374] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.875104 2187 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.875741 2186 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:35.875784 2187 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.876551 2186 ts_tablet_manager.cc:1397] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.002s
I20260927 19:37:35.878557 2187 ts_tablet_manager.cc:1428] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 19:37:35.879654 2187 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.886359 2187 tablet_bootstrap.cc:654] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.890447 1892 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:35.897248 2216 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.897791 2216 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.897724 2186 raft_consensus.cc:348] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.898783 2186 raft_consensus.cc:374] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.899120 2186 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.900038 2216 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.901455 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.901517 2186 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.902190 1996 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:35.902391 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:35.903466 2111 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:37:35.904104 2110 leader_election.cc:341] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.904238 2216 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:35.904592 2216 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:35.904940 2216 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.916409 2216 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.917822 2186 ts_tablet_manager.cc:1428] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260927 19:37:35.920645 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:35.921056 2215 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.922075 2220 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.921782 2215 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.922726 2220 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } }
I20260927 19:37:35.926759 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.927567 2215 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.928229 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:35.928747 2037 leader_election.cc:341] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.929490 1996 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.932281 2187 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:35.932757 2185 tablet_bootstrap.cc:492] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:35.932741 2187 ts_tablet_manager.cc:1397] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.029s
I20260927 19:37:35.933312 2185 ts_tablet_manager.cc:1397] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.075s user 0.032s sys 0.027s
I20260927 19:37:35.934255 2220 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:37:35.934525 2037 leader_election.cc:341] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.934993 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:35.935554 2216 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.928993 2186 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:35.936990 2187 raft_consensus.cc:348] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.937248 2185 raft_consensus.cc:348] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.937781 2187 raft_consensus.cc:374] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.937887 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.938035 2185 raft_consensus.cc:374] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:35.938169 2187 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.938411 2185 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:35.938954 2187 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.939116 2185 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.939600 2215 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.940212 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:35.941172 2187 ts_tablet_manager.cc:1428] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 19:37:35.940960 2069 raft_consensus.cc:2379] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:37:35.942442 2187 tablet_bootstrap.cc:492] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:35.942528 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:35.943601 2186 tablet_bootstrap.cc:654] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.944582 2110 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:35.948495 2187 tablet_bootstrap.cc:654] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:35.950078 2216 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:35.954648 2215 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.955147 2215 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.957093 2215 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:35.966516 2185 ts_tablet_manager.cc:1428] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260927 19:37:35.967317 1996 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:35.969084 2111 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:35.970353 2220 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:35.971741 2220 raft_consensus.cc:686] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:35.972810 2220 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:35.977972 2110 leader_election.cc:341] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.978238 2185 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:35.979527 2253 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:35.979992 2253 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:35.980850 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:35.981498 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:35.983263 2253 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:35.984179 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:35.986131 2111 leader_election.cc:341] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:35.987071 2145 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:35.981503 1995 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:35.988852 2216 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:35.990293 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:35.991732 2215 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:35.992272 2215 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:35.992755 2215 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.000526 2185 tablet_bootstrap.cc:654] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.005559 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:36.006798 2110 leader_election.cc:341] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.007272 2110 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:36.008057 2186 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.008673 2186 ts_tablet_manager.cc:1397] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.080s user 0.027s sys 0.041s
I20260927 19:37:36.008982 2220 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.012482 2186 raft_consensus.cc:348] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.013566 2186 raft_consensus.cc:374] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.013988 2186 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.015021 2186 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.013695 2187 tablet_bootstrap.cc:492] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.014086 2215 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.017128 2187 ts_tablet_manager.cc:1397] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.075s user 0.020s sys 0.025s
I20260927 19:37:36.018783 2220 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.009254 1891 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:36.020979 2186 ts_tablet_manager.cc:1428] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20260927 19:37:36.021953 2186 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.022184 2187 raft_consensus.cc:348] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.022769 2185 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.027892 2186 tablet_bootstrap.cc:654] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.022918 2187 raft_consensus.cc:374] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.028563 2187 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.029510 2185 ts_tablet_manager.cc:1397] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.014s
I20260927 19:37:36.029170 2187 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.031603 2187 ts_tablet_manager.cc:1428] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260927 19:37:36.032467 2187 tablet_bootstrap.cc:492] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.033483 2186 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.033913 2256 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.035126 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.035802 1995 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.034574 2256 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.038144 2185 raft_consensus.cc:348] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.042212 2187 tablet_bootstrap.cc:654] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.042131 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.042768 2145 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.038823 2185 raft_consensus.cc:374] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.044937 2185 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.045620 2185 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.047257 2256 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.047914 2185 ts_tablet_manager.cc:1428] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 19:37:36.034574 2215 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.049031 2185 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.051782 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.053112 2186 ts_tablet_manager.cc:1397] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.003s
I20260927 19:37:36.054960 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.055718 1994 raft_consensus.cc:2379] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 1: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:36.056573 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.057869 2256 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.063385 1995 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.065508 2185 tablet_bootstrap.cc:654] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.067793 2256 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.067140 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.069049 2145 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.071446 2186 raft_consensus.cc:348] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.072211 2186 raft_consensus.cc:374] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.072387 2186 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.072794 2186 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.071447 2256 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.075196 2186 ts_tablet_manager.cc:1428] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260927 19:37:36.076007 2186 tablet_bootstrap.cc:492] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.076538 2187 tablet_bootstrap.cc:492] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.077528 2187 ts_tablet_manager.cc:1397] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.045s user 0.018s sys 0.013s
I20260927 19:37:36.080226 2187 raft_consensus.cc:348] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.080932 2187 raft_consensus.cc:374] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.081192 2187 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.084332 2186 tablet_bootstrap.cc:654] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.081923 2187 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.085736 2190 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.086220 2190 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.087002 2187 ts_tablet_manager.cc:1428] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 19:37:36.088126 2190 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.090234 2187 tablet_bootstrap.cc:492] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.091876 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:36.092739 2069 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:36.094051 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.094844 2145 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:36.100389 2185 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.100559 2069 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.100849 2185 ts_tablet_manager.cc:1397] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.001s
I20260927 19:37:36.102694 1962 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:36.103543 2190 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.104024 2190 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.104439 2190 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.104357 2185 raft_consensus.cc:348] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.105167 2185 raft_consensus.cc:374] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.105445 2185 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.112529 2185 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.114418 2256 raft_consensus.cc:686] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.115222 2185 ts_tablet_manager.cc:1428] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.014s user 0.008s sys 0.001s
I20260927 19:37:36.116066 2185 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.116355 2186 tablet_bootstrap.cc:492] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.116528 2216 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.116855 2186 ts_tablet_manager.cc:1397] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.013s
I20260927 19:37:36.119959 2186 raft_consensus.cc:348] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.120719 2186 raft_consensus.cc:374] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.118104 2190 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.121003 2186 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.121711 2186 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.118139 2256 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.124619 2186 ts_tablet_manager.cc:1428] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 19:37:36.125555 2186 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.125864 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:36.126503 2070 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.127911 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.128559 2145 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.128520 2190 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.132762 2186 tablet_bootstrap.cc:654] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.133136 2215 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.133994 2070 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:36.133723 2215 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.135015 1961 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.136238 2215 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.137286 2190 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.137992 2190 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:36.138195 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.138832 1995 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:36.138876 2190 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.140545 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.141202 2144 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
W20260927 19:37:36.144071 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:36.144716 2185 tablet_bootstrap.cc:654] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.147121 2145 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:36.144717 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:36.150063 2256 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.150523 2256 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.150892 2256 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.154152 2185 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.154000 1893 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.155027 2185 ts_tablet_manager.cc:1397] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.009s
I20260927 19:37:36.156350 2187 tablet_bootstrap.cc:654] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.157928 2256 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.161008 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.161901 1995 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.164359 2185 raft_consensus.cc:348] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.165891 2190 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.165079 2185 raft_consensus.cc:374] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.169595 2185 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.169107 1995 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.170832 2185 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.174485 2186 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.174880 2186 ts_tablet_manager.cc:1397] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.008s
I20260927 19:37:36.176580 2186 raft_consensus.cc:348] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.177095 2186 raft_consensus.cc:374] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.177389 2186 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.171600 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.178038 2186 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.178881 2185 ts_tablet_manager.cc:1428] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.024s user 0.008s sys 0.003s
I20260927 19:37:36.180055 2185 tablet_bootstrap.cc:492] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.181178 2262 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.188670 2186 ts_tablet_manager.cc:1428] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260927 19:37:36.189723 2186 tablet_bootstrap.cc:492] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.189558 2262 raft_consensus.cc:686] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.192395 2262 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.197377 2256 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.198030 2216 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.198050 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.199285 2144 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.198721 2216 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.200402 2186 tablet_bootstrap.cc:654] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.202405 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:36.206822 2111 leader_election.cc:341] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.207055 2144 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.210381 2216 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:36.208060 1891 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.211967 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:36.214468 2110 leader_election.cc:341] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.215027 2110 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:36.216722 2185 tablet_bootstrap.cc:654] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.216735 2216 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.221030 2187 tablet_bootstrap.cc:492] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.221562 2187 ts_tablet_manager.cc:1397] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.134s user 0.021s sys 0.011s
I20260927 19:37:36.224538 2187 raft_consensus.cc:348] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.229940 2187 raft_consensus.cc:374] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.230268 2187 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.231415 2187 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.231985 2186 tablet_bootstrap.cc:492] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.232474 2186 ts_tablet_manager.cc:1397] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.004s
I20260927 19:37:36.233500 2187 ts_tablet_manager.cc:1428] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260927 19:37:36.234396 2187 tablet_bootstrap.cc:492] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.234781 2186 raft_consensus.cc:348] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.235472 2186 raft_consensus.cc:374] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.235719 2186 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.236404 2186 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.238787 2186 ts_tablet_manager.cc:1428] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 19:37:36.239878 2186 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.239493 1892 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.247112 2187 tablet_bootstrap.cc:654] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.248140 2186 tablet_bootstrap.cc:654] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.253036 2211 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.261705 2211 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.262202 2211 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.265766 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.266650 2145 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:36.268338 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:36.269208 2213 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.269587 2213 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.269901 2213 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.274648 2211 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.275015 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:36.277156 1961 leader_election.cc:341] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.284513 2213 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.288535 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:36.289091 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.289906 2145 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:36.290048 1961 leader_election.cc:341] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.296890 2145 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:36.302804 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:37:36.304530 2244 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.304795 2216 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.304996 2244 raft_consensus.cc:686] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:36.305234 2216 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.305860 2244 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.307399 2216 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:36.308817 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:36.308838 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.310127 2070 raft_consensus.cc:2379] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:37:36.311710 2110 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:36.313067 2216 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.322098 2216 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.326010 2262 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.327204 2262 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.329133 2262 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.329630 1893 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.329629 2213 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.336009 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.337466 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.338104 1995 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
W20260927 19:37:36.339425 2037 leader_election.cc:341] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.345798 2186 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.346292 2186 ts_tablet_manager.cc:1397] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.107s user 0.031s sys 0.047s
I20260927 19:37:36.343452 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.348063 2262 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.348428 2262 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.348706 2262 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.351389 2186 raft_consensus.cc:348] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.351856 2269 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.352501 2186 raft_consensus.cc:374] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.352859 2186 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.352793 2269 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.353791 2186 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.355014 2269 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.356178 2186 ts_tablet_manager.cc:1428] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260927 19:37:36.356563 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.357198 2186 tablet_bootstrap.cc:492] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.357558 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:36.359475 2037 leader_election.cc:341] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.357205 1995 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:36.360745 2216 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.361589 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:37:36.361815 1962 consensus_peers.cc:487] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Couldn't send request to peer 69f09866d0a1477585f05b2f82ff4c67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:36.363634 2186 tablet_bootstrap.cc:654] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.365667 2269 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.366201 2269 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.365722 2262 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.366508 2269 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.368448 2262 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.370453 2185 tablet_bootstrap.cc:492] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.370918 2185 ts_tablet_manager.cc:1397] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.191s user 0.049s sys 0.042s
I20260927 19:37:36.370579 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.371449 1996 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.373411 2185 raft_consensus.cc:348] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.374130 2185 raft_consensus.cc:374] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.374374 2185 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.375074 2185 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.375975 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.377141 2185 ts_tablet_manager.cc:1428] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260927 19:37:36.381695 2037 leader_election.cc:341] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.381976 2187 tablet_bootstrap.cc:492] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.382388 2187 ts_tablet_manager.cc:1397] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.148s user 0.023s sys 0.009s
I20260927 19:37:36.384323 1996 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.386049 2185 tablet_bootstrap.cc:492] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.385958 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.386766 2262 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.387269 2186 tablet_bootstrap.cc:492] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.387485 2262 raft_consensus.cc:686] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.390494 2186 ts_tablet_manager.cc:1397] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.014s
I20260927 19:37:36.387224 2220 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260927 19:37:36.391302 2262 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.392938 2186 raft_consensus.cc:348] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.393653 2186 raft_consensus.cc:374] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.393908 2186 raft_consensus.cc:729] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.394492 2186 consensus_queue.cc:260] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.395411 2185 tablet_bootstrap.cc:654] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.396729 2186 ts_tablet_manager.cc:1428] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 19:37:36.397601 2186 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.404659 2186 tablet_bootstrap.cc:654] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.410727 2269 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.390093 2187 raft_consensus.cc:348] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.413342 2269 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.414047 2187 raft_consensus.cc:374] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.414846 2187 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.415499 2187 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.416196 2185 tablet_bootstrap.cc:492] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.416630 2185 ts_tablet_manager.cc:1397] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.005s
I20260927 19:37:36.417814 2187 ts_tablet_manager.cc:1428] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.035s user 0.005s sys 0.003s
I20260927 19:37:36.418644 2187 tablet_bootstrap.cc:492] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.418996 2185 raft_consensus.cc:348] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.419667 2185 raft_consensus.cc:374] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.419895 2185 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.420665 2185 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.423471 2185 ts_tablet_manager.cc:1428] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:36.424341 2185 tablet_bootstrap.cc:492] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.424667 2187 tablet_bootstrap.cc:654] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.428996 1892 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.431459 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.433733 1995 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.435452 2185 tablet_bootstrap.cc:654] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.437721 2187 tablet_bootstrap.cc:492] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.438707 2213 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.438192 2187 ts_tablet_manager.cc:1397] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.011s
I20260927 19:37:36.441598 1995 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.432914 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.443334 2187 raft_consensus.cc:348] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.443845 2213 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.444804 2187 raft_consensus.cc:374] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.445200 2187 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.445879 2187 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.447505 2187 ts_tablet_manager.cc:1428] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 19:37:36.448205 2187 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
W20260927 19:37:36.450969 2037 leader_election.cc:341] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.431720 2186 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.452467 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.453253 2186 ts_tablet_manager.cc:1397] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.016s
I20260927 19:37:36.453557 2269 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.454135 2269 raft_consensus.cc:686] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.454916 2269 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.456252 2186 raft_consensus.cc:348] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.457129 2186 raft_consensus.cc:374] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.457449 2186 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.458133 2186 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.460387 2186 ts_tablet_manager.cc:1428] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20260927 19:37:36.460412 2185 tablet_bootstrap.cc:492] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.461148 2185 ts_tablet_manager.cc:1397] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.009s
I20260927 19:37:36.461284 2187 tablet_bootstrap.cc:654] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.461486 2186 tablet_bootstrap.cc:492] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.464977 2185 raft_consensus.cc:348] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.466223 2185 raft_consensus.cc:374] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.466732 2185 raft_consensus.cc:729] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.467708 2185 consensus_queue.cc:260] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.469130 2186 tablet_bootstrap.cc:654] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.474179 2185 ts_tablet_manager.cc:1428] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260927 19:37:36.479918 2262 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.484272 2190 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.485095 2262 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.487397 2262 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.488674 2185 tablet_bootstrap.cc:492] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.490952 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.495081 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.495776 1995 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:36.497175 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.497584 2185 tablet_bootstrap.cc:654] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.498063 2269 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260927 19:37:36.498186 2037 leader_election.cc:341] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.498502 2269 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.504438 2269 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.514207 2187 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.514866 2187 ts_tablet_manager.cc:1397] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.002s
I20260927 19:37:36.528141 2187 raft_consensus.cc:348] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.529244 2187 raft_consensus.cc:374] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.529726 2187 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.531117 2269 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.533058 2187 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.535181 1891 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.536386 2187 ts_tablet_manager.cc:1428] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.021s user 0.010s sys 0.000s
I20260927 19:37:36.537518 2187 tablet_bootstrap.cc:492] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.537916 2186 tablet_bootstrap.cc:492] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.538362 2186 ts_tablet_manager.cc:1397] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.077s user 0.023s sys 0.033s
I20260927 19:37:36.540307 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.540948 1995 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.541599 2186 raft_consensus.cc:348] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.542085 2269 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.542317 2186 raft_consensus.cc:374] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.542629 2186 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.543244 2186 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.545202 2185 tablet_bootstrap.cc:492] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.545710 2185 ts_tablet_manager.cc:1397] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.057s user 0.020s sys 0.020s
I20260927 19:37:36.545691 2186 ts_tablet_manager.cc:1428] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 19:37:36.546737 2186 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.547096 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.548369 2185 raft_consensus.cc:348] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.549137 2185 raft_consensus.cc:374] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.549511 2185 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.550406 2262 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.550206 2185 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:36.552520 2037 leader_election.cc:341] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.553193 2186 tablet_bootstrap.cc:654] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.554226 2216 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.554726 2216 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.556757 2185 ts_tablet_manager.cc:1428] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260927 19:37:36.557955 2185 tablet_bootstrap.cc:492] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.558777 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.559168 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:36.560258 2111 leader_election.cc:341] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:36.560925 2110 leader_election.cc:341] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.562047 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:36.563129 2220 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.556833 2216 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:36.564404 2185 tablet_bootstrap.cc:654] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.569303 2262 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.570058 2262 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.572916 2262 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.575376 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.583923 1996 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:36.575459 1995 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.577502 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.585103 2144 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:36.577718 2269 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.587469 2256 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.587976 2256 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.588822 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:36.590118 2256 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.590730 2256 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.591127 2256 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.591655 2256 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.590889 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.593343 2215 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.593961 2215 raft_consensus.cc:686] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.581171 2187 tablet_bootstrap.cc:654] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.595038 2215 consensus_queue.cc:237] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.595747 2213 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.611636 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.621026 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.622196 2144 raft_consensus.cc:2379] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 1: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:37:36.626335 2187 tablet_bootstrap.cc:492] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.626935 2187 ts_tablet_manager.cc:1397] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.090s user 0.026s sys 0.001s
I20260927 19:37:36.627888 2269 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.629801 2187 raft_consensus.cc:348] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.630571 2187 raft_consensus.cc:374] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.630859 2187 raft_consensus.cc:729] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.632658 2187 consensus_queue.cc:260] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.636476 2187 ts_tablet_manager.cc:1428] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260927 19:37:36.637881 2187 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.638728 2037 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.639825 2262 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.640745 1892 catalog_manager.cc:5495] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.655130 2269 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.655840 2269 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.659298 2269 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.662797 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.663480 1996 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:36.645789 2256 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.673399 2185 tablet_bootstrap.cc:492] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.674216 2185 ts_tablet_manager.cc:1397] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.013s
I20260927 19:37:36.677603 2186 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.678032 2186 ts_tablet_manager.cc:1397] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.131s user 0.040s sys 0.030s
I20260927 19:37:36.678614 2220 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.679127 2220 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.681113 2186 raft_consensus.cc:348] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.681818 2186 raft_consensus.cc:374] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.682076 2186 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.674880 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.682673 2186 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.682941 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.683856 1996 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:37:36.684945 2186 ts_tablet_manager.cc:1428] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260927 19:37:36.685926 2186 tablet_bootstrap.cc:492] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.687018 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.678243 2185 raft_consensus.cc:348] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.688871 2185 raft_consensus.cc:374] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.689604 2220 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:36.689170 2185 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.690845 2269 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.691054 2256 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.691179 2185 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.693565 2186 tablet_bootstrap.cc:654] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.693673 2269 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.694103 2269 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.695137 2190 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.697319 2185 ts_tablet_manager.cc:1428] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260927 19:37:36.698614 2185 tablet_bootstrap.cc:492] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.699004 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.699738 2145 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:36.705312 2037 leader_election.cc:341] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.711591 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.713601 2269 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.713946 1995 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.717824 2187 tablet_bootstrap.cc:654] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.718385 2145 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.722363 2262 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.722918 2185 tablet_bootstrap.cc:654] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.723969 2186 tablet_bootstrap.cc:492] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.724454 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:36.724462 2186 ts_tablet_manager.cc:1397] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.011s
I20260927 19:37:36.727279 2269 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.729182 2186 raft_consensus.cc:348] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.730051 2186 raft_consensus.cc:374] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.730363 2186 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.736110 1995 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.740545 2186 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.742370 2110 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:36.743783 2186 ts_tablet_manager.cc:1428] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260927 19:37:36.743716 2220 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.754578 2220 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:36.755990 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.758312 2186 tablet_bootstrap.cc:492] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.758445 2269 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.759702 2269 raft_consensus.cc:686] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.761029 2269 consensus_queue.cc:237] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.765738 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:36.766482 1995 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.768175 2187 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.768533 2186 tablet_bootstrap.cc:654] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.768769 2187 ts_tablet_manager.cc:1397] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.131s user 0.035s sys 0.010s
I20260927 19:37:36.771279 2187 raft_consensus.cc:348] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.771469 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.772970 2187 raft_consensus.cc:374] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.773307 2187 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.774158 1995 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:36.774667 2187 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.776363 2145 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.780455 2186 tablet_bootstrap.cc:492] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.780913 2186 ts_tablet_manager.cc:1397] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.007s
I20260927 19:37:36.782235 2187 ts_tablet_manager.cc:1428] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 19:37:36.783253 2187 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.783430 2186 raft_consensus.cc:348] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.784099 2186 raft_consensus.cc:374] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.784366 2186 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.784974 2186 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.787585 2186 ts_tablet_manager.cc:1428] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 19:37:36.788460 2186 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.789510 2187 tablet_bootstrap.cc:654] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.794303 2186 tablet_bootstrap.cc:654] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.805775 2187 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.806380 2187 ts_tablet_manager.cc:1397] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.001s
I20260927 19:37:36.809836 2187 raft_consensus.cc:348] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.810562 2187 raft_consensus.cc:374] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.810803 2187 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.811349 2187 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.813431 2187 ts_tablet_manager.cc:1428] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:36.813805 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.814332 2187 tablet_bootstrap.cc:492] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.814692 2262 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.815167 2262 raft_consensus.cc:686] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:36.816058 2190 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.816506 2190 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.818468 2215 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 19:37:36.819799 2037 leader_election.cc:341] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.816864 2262 consensus_queue.cc:237] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.822970 2143 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.824849 2187 tablet_bootstrap.cc:654] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.829787 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.830730 2144 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:36.832574 1892 catalog_manager.cc:5495] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.833844 1962 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:36.834684 2211 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.835170 2211 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.835510 2211 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.825212 2190 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.845842 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:36.846534 2069 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:36.847061 2256 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.845471 2215 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:37:36.847769 2256 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.850242 2256 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.855885 2211 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.860908 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:36.859771 2211 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.861974 2144 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.866894 2186 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.867527 2186 ts_tablet_manager.cc:1397] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.007s
I20260927 19:37:36.868536 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.870682 2186 raft_consensus.cc:348] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.871691 2186 raft_consensus.cc:374] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.872011 2186 raft_consensus.cc:729] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.870119 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.873317 2186 consensus_queue.cc:260] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.874648 2262 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.875509 2262 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.876541 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:36.877112 2069 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.877561 2262 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.879944 2262 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 19:37:36.880430 2037 leader_election.cc:341] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.882858 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.884783 2069 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:36.881395 2262 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260927 19:37:36.888686 2186 ts_tablet_manager.cc:1428] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20260927 19:37:36.889974 2186 tablet_bootstrap.cc:492] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.897210 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.897938 2143 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
W20260927 19:37:36.898960 2037 leader_election.cc:341] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.899405 2037 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:36.900159 2262 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 19:37:36.900650 2037 leader_election.cc:341] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.907867 2211 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:36.913420 1961 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:36.914291 2211 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:36.914942 2211 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:36.915282 2185 tablet_bootstrap.cc:492] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:36.915704 2185 ts_tablet_manager.cc:1397] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.217s user 0.038s sys 0.034s
I20260927 19:37:36.915766 2211 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.918150 2185 raft_consensus.cc:348] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.918172 2144 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:36.918851 2185 raft_consensus.cc:374] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.919126 2185 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.920235 2186 tablet_bootstrap.cc:654] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:36.919708 2185 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.926287 2262 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.926824 2262 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.927531 2185 ts_tablet_manager.cc:1428] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260927 19:37:36.927481 1891 catalog_manager.cc:5495] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:36.928447 2262 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.932523 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:36.928515 2185 tablet_bootstrap.cc:492] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:36.936627 2187 tablet_bootstrap.cc:492] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:36.937259 2187 ts_tablet_manager.cc:1397] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.009s
I20260927 19:37:36.938603 2262 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:36.939100 2262 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:36.939500 2262 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:36.940097 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.949949 2253 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:36.957593 2186 tablet_bootstrap.cc:492] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:36.958241 2186 ts_tablet_manager.cc:1397] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.006s
I20260927 19:37:36.962931 2186 raft_consensus.cc:348] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.964787 2186 raft_consensus.cc:374] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.965090 2186 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.965900 2186 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.967330 2186 ts_tablet_manager.cc:1428] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 19:37:36.968195 2186 tablet_bootstrap.cc:492] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:36.957808 2253 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.973778 2253 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:36.972100 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:36.979627 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:36.980410 1994 raft_consensus.cc:2379] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:37:36.984158 2187 raft_consensus.cc:348] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.984836 2187 raft_consensus.cc:374] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:36.985069 2187 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:36.985749 2187 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:36.974304 2037 leader_election.cc:341] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:36.991839 2187 ts_tablet_manager.cc:1428] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260927 19:37:36.991447 2262 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:36.992630 2187 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:36.993453 2262 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:36.997852 2190 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:36.987051 1891 catalog_manager.cc:5495] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.001854 2185 tablet_bootstrap.cc:654] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.002566 2190 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.003118 2190 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.006726 2187 tablet_bootstrap.cc:654] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.008282 2186 tablet_bootstrap.cc:654] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.013284 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:37.016619 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.018086 2190 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:37:37.018632 1962 leader_election.cc:341] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.021728 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:37.022406 2142 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.030864 2185 tablet_bootstrap.cc:492] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.031175 2142 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:37.032351 2186 tablet_bootstrap.cc:492] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.032869 2185 ts_tablet_manager.cc:1397] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.105s user 0.011s sys 0.009s
I20260927 19:37:37.033191 2186 ts_tablet_manager.cc:1397] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.004s
W20260927 19:37:37.033494 2037 leader_election.cc:341] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.035887 2185 raft_consensus.cc:348] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.036295 2186 raft_consensus.cc:348] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.036696 2185 raft_consensus.cc:374] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.036947 2186 raft_consensus.cc:374] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.037128 2185 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.037249 2186 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.037994 2186 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.040050 2186 ts_tablet_manager.cc:1428] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260927 19:37:37.040916 2186 tablet_bootstrap.cc:492] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.034108 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.045110 2185 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.046344 2186 tablet_bootstrap.cc:654] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.047535 2190 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.047741 2185 ts_tablet_manager.cc:1428] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260927 19:37:37.048004 2190 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.049078 2185 tablet_bootstrap.cc:492] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.050020 2190 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.051904 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:37.053807 1962 leader_election.cc:341] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.056172 2185 tablet_bootstrap.cc:654] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.056631 2269 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.063187 2186 tablet_bootstrap.cc:492] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.063761 2186 ts_tablet_manager.cc:1397] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20260927 19:37:37.066280 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.067447 2256 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.067935 2256 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:37.067663 2186 raft_consensus.cc:348] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.068363 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.068364 2186 raft_consensus.cc:374] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.069722 2186 raft_consensus.cc:729] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.068825 2256 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.070338 2186 consensus_queue.cc:260] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.072307 2186 ts_tablet_manager.cc:1428] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260927 19:37:37.073252 2186 tablet_bootstrap.cc:492] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.078217 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.079893 2070 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.079816 2262 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 19:37:37.079972 2037 leader_election.cc:341] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.080940 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.081799 2071 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.084969 2186 tablet_bootstrap.cc:654] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.086901 2187 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.087563 2187 ts_tablet_manager.cc:1397] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.095s user 0.023s sys 0.032s
I20260927 19:37:37.090704 2185 tablet_bootstrap.cc:492] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.090842 2187 raft_consensus.cc:348] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.093951 2142 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.095937 2110 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:37.098191 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.119196 2213 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.119683 2213 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.119974 2213 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.110453 2269 consensus_queue.cc:1035] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:37.119092 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.106900 2215 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.122269 2190 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.122664 2190 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.123090 2190 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.122704 2215 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.124930 2215 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.125489 2187 raft_consensus.cc:374] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.125921 2187 raft_consensus.cc:729] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.126988 2187 consensus_queue.cc:260] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.130824 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.132905 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.138180 1892 catalog_manager.cc:5495] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.139613 2187 ts_tablet_manager.cc:1428] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.052s user 0.009s sys 0.001s
I20260927 19:37:37.140507 2187 tablet_bootstrap.cc:492] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
W20260927 19:37:37.146263 2037 leader_election.cc:341] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:37.147075 2037 leader_election.cc:341] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.147469 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.148744 2185 ts_tablet_manager.cc:1397] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.016s
I20260927 19:37:37.151262 2185 raft_consensus.cc:348] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.151930 2185 raft_consensus.cc:374] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.152191 2185 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.153513 2211 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.153913 2211 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.154313 2216 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.154740 2216 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.155789 2211 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.155895 2256 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.156576 2216 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.157035 2216 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.157271 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.157467 2253 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.157889 2143 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.158473 2256 consensus_queue.cc:1035] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.159116 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.159929 2211 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.160259 2211 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.160575 2211 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.160760 2216 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.161818 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.162474 2071 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:37.163616 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.163728 2216 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.164373 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.165154 2216 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.165568 2216 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.165895 2216 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.165795 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.166131 2186 tablet_bootstrap.cc:492] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.166682 2186 ts_tablet_manager.cc:1397] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.094s user 0.027s sys 0.022s
W20260927 19:37:37.167447 2111 leader_election.cc:341] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:37.168097 2111 leader_election.cc:341] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.169376 2186 raft_consensus.cc:348] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.170212 2186 raft_consensus.cc:374] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.170535 2186 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.171042 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.171187 2186 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.173563 2186 ts_tablet_manager.cc:1428] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:37.174472 2186 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.174373 2190 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.174700 2211 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.152810 2185 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.176131 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.177151 2071 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.178179 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:37:37.178297 2110 leader_election.cc:341] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.181968 2186 tablet_bootstrap.cc:654] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.178807 2110 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:37.184657 2220 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.185628 2185 ts_tablet_manager.cc:1428] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.037s user 0.005s sys 0.002s
I20260927 19:37:37.185711 2142 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.186111 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:37.186652 2185 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
W20260927 19:37:37.187575 1962 leader_election.cc:341] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.188380 2190 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.178753 2213 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.189862 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:37.190459 2071 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.197791 2186 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.198544 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:37.199898 2070 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.201234 2186 ts_tablet_manager.cc:1397] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.010s
I20260927 19:37:37.201767 2216 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.204952 2186 raft_consensus.cc:348] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.206506 2186 raft_consensus.cc:374] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.206925 2186 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.208425 2142 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.207796 2186 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.181247 2187 tablet_bootstrap.cc:654] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.221827 2213 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.214036 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:37:37.227193 2111 leader_election.cc:341] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.217202 2211 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.211331 2070 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.219004 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:37.230041 2069 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:37.229432 2037 consensus_peers.cc:487] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 -> Peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Couldn't send request to peer 69f09866d0a1477585f05b2f82ff4c67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:37.231761 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:37.232758 2186 ts_tablet_manager.cc:1428] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260927 19:37:37.213266 2071 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.233810 2186 tablet_bootstrap.cc:492] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.234499 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.234964 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.235610 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:37.236428 2333 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.236878 2333 raft_consensus.cc:686] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:37.237771 2333 consensus_queue.cc:237] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.238917 2185 tablet_bootstrap.cc:654] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.241549 2186 tablet_bootstrap.cc:654] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.241585 2190 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.242040 2216 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.242236 2190 raft_consensus.cc:686] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:37.243499 2190 consensus_queue.cc:237] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.247779 2069 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.251680 2269 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.252146 2269 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.233738 2213 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.253006 2187 tablet_bootstrap.cc:492] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.253609 2187 ts_tablet_manager.cc:1397] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.113s user 0.014s sys 0.002s
I20260927 19:37:37.257601 2213 raft_consensus.cc:686] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:37.263569 2269 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.270642 2185 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.271404 2185 ts_tablet_manager.cc:1397] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.004s
I20260927 19:37:37.273442 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:37:37.274359 1962 leader_election.cc:341] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.259332 2213 consensus_queue.cc:237] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.275514 2187 raft_consensus.cc:348] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.278750 2187 raft_consensus.cc:374] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.279466 2187 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.280686 2187 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.279983 2069 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.284744 2186 tablet_bootstrap.cc:492] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.285185 2186 ts_tablet_manager.cc:1397] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.006s
I20260927 19:37:37.290109 2187 ts_tablet_manager.cc:1428] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.036s user 0.008s sys 0.000s
I20260927 19:37:37.291013 2187 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.296110 2186 raft_consensus.cc:348] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.297004 2186 raft_consensus.cc:374] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.297334 2186 raft_consensus.cc:729] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.298388 2186 consensus_queue.cc:260] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.307878 2215 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:37.310118 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.315243 2244 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.318444 2186 ts_tablet_manager.cc:1428] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.033s user 0.001s sys 0.007s
I20260927 19:37:37.320176 2186 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.322479 2338 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:37:37.322834 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:37.323194 2338 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.333488 2338 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.334796 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.335255 2069 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:37.338866 2185 raft_consensus.cc:348] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.339296 2187 tablet_bootstrap.cc:654] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.339475 2185 raft_consensus.cc:374] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.339741 2185 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.341388 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.344871 2185 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.342180 2142 raft_consensus.cc:2379] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 1: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:37:37.345778 2333 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.347157 2333 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.349285 2185 ts_tablet_manager.cc:1428] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.078s user 0.008s sys 0.001s
I20260927 19:37:37.351974 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:37.359622 2333 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.360082 1892 catalog_manager.cc:5495] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.367231 1892 catalog_manager.cc:5495] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.368750 2216 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.368841 1892 catalog_manager.cc:5495] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.370123 2216 raft_consensus.cc:686] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:37.370980 2216 consensus_queue.cc:237] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:37.366014 2037 leader_election.cc:341] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.372552 2185 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.374642 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.375247 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:37.376842 1962 leader_election.cc:341] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.377060 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.377744 2071 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.377140 2186 tablet_bootstrap.cc:654] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.379976 1961 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.380910 2213 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.381737 2338 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.385325 1961 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.386668 2244 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.387111 2244 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.387446 2244 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.403663 2185 tablet_bootstrap.cc:654] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.410969 2338 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260927 19:37:37.411865 2037 leader_election.cc:341] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.412564 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.421507 2262 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.423930 2262 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:37.430022 2244 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.430676 2187 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.431318 2187 ts_tablet_manager.cc:1397] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.140s user 0.013s sys 0.028s
I20260927 19:37:37.433314 1892 catalog_manager.cc:5495] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:37.433903 2187 raft_consensus.cc:348] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.434700 2187 raft_consensus.cc:374] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.435004 2187 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.435706 2262 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:37.435761 2187 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.437690 2244 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.438544 2187 ts_tablet_manager.cc:1428] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260927 19:37:37.442363 2187 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.449584 2186 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.450197 2186 ts_tablet_manager.cc:1397] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.130s user 0.007s sys 0.006s
I20260927 19:37:37.452742 2186 raft_consensus.cc:348] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.455145 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:37.458524 2070 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.454378 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:37:37.461251 1962 leader_election.cc:341] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.465319 2070 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.466919 2185 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.468248 2185 ts_tablet_manager.cc:1397] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.010s
I20260927 19:37:37.472721 2185 raft_consensus.cc:348] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.473356 2185 raft_consensus.cc:374] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.473613 2185 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.474148 2185 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.475641 2185 ts_tablet_manager.cc:1428] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 19:37:37.476032 2186 raft_consensus.cc:374] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.476259 2186 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.476832 2186 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.478410 1961 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.478902 2186 ts_tablet_manager.cc:1428] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20260927 19:37:37.479775 2186 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.480125 2338 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.480706 2338 raft_consensus.cc:686] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:37.482128 2338 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.483275 2185 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.481292 2187 tablet_bootstrap.cc:654] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.493515 2185 tablet_bootstrap.cc:654] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.497570 2186 tablet_bootstrap.cc:654] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.509965 1891 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.515762 2186 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.521850 2185 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.522333 2185 ts_tablet_manager.cc:1397] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.002s
I20260927 19:37:37.523239 2187 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.523800 2187 ts_tablet_manager.cc:1397] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.009s
I20260927 19:37:37.525166 2185 raft_consensus.cc:348] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.525879 2185 raft_consensus.cc:374] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.526144 2185 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.526551 2187 raft_consensus.cc:348] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.527122 2187 raft_consensus.cc:374] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.526741 2185 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.527383 2187 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.528481 2187 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.528966 2185 ts_tablet_manager.cc:1428] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260927 19:37:37.516258 2186 ts_tablet_manager.cc:1397] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.005s
I20260927 19:37:37.530542 2187 ts_tablet_manager.cc:1428] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260927 19:37:37.531472 2185 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.534019 2187 tablet_bootstrap.cc:492] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.537810 2185 tablet_bootstrap.cc:654] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.542366 2187 tablet_bootstrap.cc:654] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.535974 2186 raft_consensus.cc:348] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.542904 2186 raft_consensus.cc:374] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.543249 2186 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.544103 2186 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.546607 2186 ts_tablet_manager.cc:1428] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20260927 19:37:37.547727 2186 tablet_bootstrap.cc:492] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.555270 2338 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.555958 2338 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.558720 2338 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.559736 2187 tablet_bootstrap.cc:492] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.560288 2187 ts_tablet_manager.cc:1397] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20260927 19:37:37.562789 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.563476 2186 tablet_bootstrap.cc:654] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.563546 2143 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.563215 2187 raft_consensus.cc:348] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.564064 2185 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.564330 2187 raft_consensus.cc:374] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.564527 2185 ts_tablet_manager.cc:1397] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.014s
I20260927 19:37:37.564725 2187 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.565860 2262 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.566030 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.566713 2069 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.566449 2262 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.568308 1962 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.568621 2262 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.568254 2185 raft_consensus.cc:348] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.569109 2262 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.569222 2185 raft_consensus.cc:374] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.569285 2338 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.569890 2338 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.569655 2262 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.570371 2338 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.571655 2262 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.572839 2187 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.579954 2185 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.579844 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.580780 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.581599 2185 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.581993 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:37.583678 2037 leader_election.cc:341] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.585493 2187 ts_tablet_manager.cc:1428] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
W20260927 19:37:37.588681 2037 leader_election.cc:341] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:37.589649 2037 leader_election.cc:341] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.590293 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.590440 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.592728 2262 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.593050 2269 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:37.595695 2187 tablet_bootstrap.cc:492] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.579862 2338 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.603646 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:37.604352 2145 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.606756 2187 tablet_bootstrap.cc:654] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.608098 2185 ts_tablet_manager.cc:1428] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
W20260927 19:37:37.609463 2037 leader_election.cc:341] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.609921 2037 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.610657 2338 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.610878 2269 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.612890 2145 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.615911 1962 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.617568 2338 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.618500 2338 raft_consensus.cc:686] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:37.619658 2338 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.608974 2185 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.628247 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:37.628918 2069 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.637949 2262 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.639336 2185 tablet_bootstrap.cc:654] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.637949 2069 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.648178 2186 tablet_bootstrap.cc:492] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.648667 2186 ts_tablet_manager.cc:1397] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.101s user 0.025s sys 0.030s
I20260927 19:37:37.651160 2186 raft_consensus.cc:348] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.651865 2186 raft_consensus.cc:374] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.653712 2186 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.654518 2186 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.656173 1892 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.667296 2186 ts_tablet_manager.cc:1428] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.018s user 0.007s sys 0.002s
I20260927 19:37:37.671363 2186 tablet_bootstrap.cc:492] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.669778 2187 tablet_bootstrap.cc:492] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.678824 2187 ts_tablet_manager.cc:1397] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.083s user 0.029s sys 0.027s
I20260927 19:37:37.681802 2187 raft_consensus.cc:348] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.682682 2187 raft_consensus.cc:374] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.683041 2187 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.683795 2187 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.685603 2186 tablet_bootstrap.cc:654] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.686244 2187 ts_tablet_manager.cc:1428] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:37.687878 2187 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.690794 2145 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.696642 2187 tablet_bootstrap.cc:654] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.697762 2185 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.698226 2185 ts_tablet_manager.cc:1397] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.089s user 0.015s sys 0.002s
I20260927 19:37:37.702353 2185 raft_consensus.cc:348] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.703508 2185 raft_consensus.cc:374] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.703888 2262 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.704483 2262 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.707355 2262 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.709726 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.711208 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.711794 2187 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.712414 2187 ts_tablet_manager.cc:1397] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.008s
W20260927 19:37:37.712724 2037 leader_election.cc:341] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:37.713317 2037 leader_election.cc:341] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.713668 2037 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.714838 2269 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.718515 2187 raft_consensus.cc:348] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.719497 2187 raft_consensus.cc:374] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.719822 2187 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.720018 2185 raft_consensus.cc:729] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.720623 2187 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.721249 2185 consensus_queue.cc:260] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.723974 2187 ts_tablet_manager.cc:1428] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260927 19:37:37.725170 2187 tablet_bootstrap.cc:492] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.732854 2187 tablet_bootstrap.cc:654] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.736409 2338 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.739451 2185 ts_tablet_manager.cc:1428] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.041s user 0.008s sys 0.000s
I20260927 19:37:37.741441 2338 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.743873 2185 tablet_bootstrap.cc:492] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.745942 2187 tablet_bootstrap.cc:492] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.746562 2187 ts_tablet_manager.cc:1397] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.011s
I20260927 19:37:37.749879 2187 raft_consensus.cc:348] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.750712 2187 raft_consensus.cc:374] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.750967 2187 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.751668 2187 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.753044 2185 tablet_bootstrap.cc:654] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.755532 2186 tablet_bootstrap.cc:492] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.756043 2186 ts_tablet_manager.cc:1397] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.085s user 0.032s sys 0.040s
I20260927 19:37:37.755003 2187 ts_tablet_manager.cc:1428] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 19:37:37.758610 2187 tablet_bootstrap.cc:492] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.758929 2216 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.759321 2186 raft_consensus.cc:348] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.759537 2216 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.759999 2186 raft_consensus.cc:374] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.760254 2186 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.760911 2186 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.761412 2216 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.762904 2186 ts_tablet_manager.cc:1428] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 19:37:37.763499 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.764214 2186 tablet_bootstrap.cc:492] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.765272 2187 tablet_bootstrap.cc:654] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.766036 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.769790 2186 tablet_bootstrap.cc:654] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.772192 2333 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.772817 2333 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.776362 2333 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.780102 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.781150 2143 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.781709 2253 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.782243 2111 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:37.782146 2253 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.783680 2220 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.784837 2145 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.785106 1962 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.786780 2244 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:37.787206 2244 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.787621 2244 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.791388 2253 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.795043 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.795924 2070 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.782794 2213 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260927 19:37:37.809680 2187 tablet_bootstrap.cc:492] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.809499 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.810164 2187 ts_tablet_manager.cc:1397] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.009s
I20260927 19:37:37.813071 2187 raft_consensus.cc:348] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.815351 2186 tablet_bootstrap.cc:492] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.815794 2186 ts_tablet_manager.cc:1397] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.011s
I20260927 19:37:37.817386 2338 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.818213 2186 raft_consensus.cc:348] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.813849 2187 raft_consensus.cc:374] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.818889 2186 raft_consensus.cc:374] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.819248 2187 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.819381 2186 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.820412 2186 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.820425 2187 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.822975 2187 ts_tablet_manager.cc:1428] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260927 19:37:37.823330 2186 ts_tablet_manager.cc:1428] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260927 19:37:37.823844 2187 tablet_bootstrap.cc:492] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:37.824458 2186 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.832775 2144 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.839248 2186 tablet_bootstrap.cc:654] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.841437 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.843410 2069 raft_consensus.cc:2379] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:37:37.847505 2110 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:37.852931 2333 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.863276 2220 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.860764 2244 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.867616 2333 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.869858 2244 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.871869 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:37.872717 2145 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.883605 2187 tablet_bootstrap.cc:654] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.886246 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:37.886870 2069 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.888753 1995 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.891819 2186 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:37.892395 2186 ts_tablet_manager.cc:1397] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.068s user 0.018s sys 0.005s
I20260927 19:37:37.906566 2145 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.907918 1962 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.908355 2069 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:37.917876 2333 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:37.919659 2333 raft_consensus.cc:686] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:37.920805 2220 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.921082 2185 tablet_bootstrap.cc:492] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:37.926033 2185 ts_tablet_manager.cc:1397] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.182s user 0.044s sys 0.044s
I20260927 19:37:37.920559 2333 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.927815 2142 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:37.930739 2185 raft_consensus.cc:348] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.931413 2185 raft_consensus.cc:374] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.931690 2185 raft_consensus.cc:729] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.934474 2244 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.935050 2244 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.937917 2269 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.938412 2269 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.940060 2244 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.940236 2269 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:37.941632 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.942348 2220 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.942857 2220 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.932258 2185 consensus_queue.cc:260] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.943312 2244 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.945581 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.945333 2186 raft_consensus.cc:348] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.946106 2186 raft_consensus.cc:374] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.946369 2186 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.946961 2186 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.948973 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.949399 2187 tablet_bootstrap.cc:492] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:37.949822 2187 ts_tablet_manager.cc:1397] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.001s
I20260927 19:37:37.949535 2253 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:37.952980 2185 ts_tablet_manager.cc:1428] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.027s user 0.004s sys 0.003s
I20260927 19:37:37.952643 2187 raft_consensus.cc:348] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.953330 2187 raft_consensus.cc:374] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:37.953606 2187 raft_consensus.cc:729] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:37.954299 2187 consensus_queue.cc:260] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.954367 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:37.956924 2071 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.957813 2185 tablet_bootstrap.cc:492] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:37.959102 2220 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.960806 2186 ts_tablet_manager.cc:1428] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.068s user 0.006s sys 0.002s
I20260927 19:37:37.942349 2145 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:37.961721 2186 tablet_bootstrap.cc:492] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:37.963538 1961 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:37.966107 2333 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:37.969622 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:37.962323 2187 ts_tablet_manager.cc:1428] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
W20260927 19:37:37.963210 2037 leader_election.cc:341] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.971750 2187 tablet_bootstrap.cc:492] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
W20260927 19:37:37.972154 2037 leader_election.cc:341] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:37.972560 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:37.975992 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.977039 2269 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:37.976847 2070 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:37.980348 2110 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:37.982738 2253 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:37.983433 2253 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:37.982331 1890 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:37.987260 2253 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:37.993717 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:37.994980 2070 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:37.977252 2333 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.001971 2185 tablet_bootstrap.cc:654] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:37.997939 2333 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:37.998898 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.009744 2333 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.011205 2253 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.001597 2220 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:37:38.011595 2253 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.011981 2253 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.016871 2220 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:37:38.017392 2220 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:38.017592 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.082295 2186 tablet_bootstrap.cc:654] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:38.092641 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 69355.6us
I20260927 19:37:38.107985 2187 tablet_bootstrap.cc:654] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.107336 2389 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:37:38.125689 2111 leader_election.cc:341] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.119971 2253 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.123950 2262 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.116580 2220 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.143989 2333 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.157213 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.158162 2070 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.158444 2388 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.159118 2388 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.149152 2262 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.166168 2144 raft_consensus.cc:1226] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:37:38.168342 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.171597 2220 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.173902 2333 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.124914 2389 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:38.173031 2037 leader_election.cc:341] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.178808 2388 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.184592 2262 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.186719 2253 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.187575 2333 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:38.190788 2389 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.191210 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:38.192085 2143 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.195976 2070 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:38.200492 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.201282 1995 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:38.211277 2185 tablet_bootstrap.cc:492] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.211794 2185 ts_tablet_manager.cc:1397] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.254s user 0.014s sys 0.000s
I20260927 19:37:38.212776 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:38.214787 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.215983 2111 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:37:38.217151 2111 leader_election.cc:341] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.240615 2387 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.241098 2387 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.242013 2387 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.217607 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:37:38.230830 2111 leader_election.cc:341] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.245250 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.218114 2185 raft_consensus.cc:348] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.256286 2185 raft_consensus.cc:374] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.256702 2185 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.257588 2185 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.238106 2333 consensus_queue.cc:1046] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.025s
W20260927 19:37:38.267719 2037 leader_election.cc:341] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.270480 2333 raft_consensus.cc:3032] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:37:38.272042 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:38.274008 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.272432 2333 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:38.275378 2269 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.275736 2333 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.276292 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.276880 2071 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.277207 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.277830 2070 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:38.262185 1961 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:38.279953 2381 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:38.285858 2187 tablet_bootstrap.cc:492] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:38.286020 2381 raft_consensus.cc:686] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:38.286321 2187 ts_tablet_manager.cc:1397] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.315s user 0.014s sys 0.022s
I20260927 19:37:38.288116 2186 tablet_bootstrap.cc:492] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:38.288537 2186 ts_tablet_manager.cc:1397] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.327s user 0.025s sys 0.030s
I20260927 19:37:38.274014 2185 ts_tablet_manager.cc:1428] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.062s user 0.011s sys 0.000s
I20260927 19:37:38.272225 2387 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.284678 2143 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:38.276885 2333 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:38.353248 2185 tablet_bootstrap.cc:492] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.352748 2187 raft_consensus.cc:348] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.353597 2187 raft_consensus.cc:374] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.353861 2187 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.354638 2187 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.357668 2187 ts_tablet_manager.cc:1428] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.071s user 0.005s sys 0.002s
I20260927 19:37:38.307968 2381 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.358731 2187 tablet_bootstrap.cc:492] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:38.360188 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:38.360842 1995 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.361075 2186 raft_consensus.cc:348] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.361780 2186 raft_consensus.cc:374] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.362053 2186 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.362740 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.362665 2186 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.364775 2186 ts_tablet_manager.cc:1428] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.076s user 0.005s sys 0.014s
I20260927 19:37:38.365144 2387 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.365998 2110 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:38.368806 2389 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 19:37:38.373101 2110 leader_election.cc:341] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.373982 2110 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:38.374809 2387 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.377581 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.378188 2066 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.380736 2186 tablet_bootstrap.cc:492] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:38.382170 1995 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:38.383776 2111 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.384897 2387 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:38.386685 2187 tablet_bootstrap.cc:654] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.385551 2387 raft_consensus.cc:686] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:38.387835 2066 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:38.390563 2387 consensus_queue.cc:237] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.400358 2185 tablet_bootstrap.cc:654] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.370617 2071 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:38.403601 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:38.404441 2389 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:38.405045 2389 raft_consensus.cc:686] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:38.406095 2389 consensus_queue.cc:237] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.413990 2244 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:38.445494 2186 tablet_bootstrap.cc:654] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:38.458088 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:38.473035 2187 tablet_bootstrap.cc:492] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:38.474130 2187 ts_tablet_manager.cc:1397] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.116s user 0.027s sys 0.014s
I20260927 19:37:38.476449 2187 raft_consensus.cc:348] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.477152 2187 raft_consensus.cc:374] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.477437 2187 raft_consensus.cc:729] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.476744 1892 catalog_manager.cc:5495] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:38.478487 2187 consensus_queue.cc:260] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.481063 2185 tablet_bootstrap.cc:492] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.481680 2338 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:38.484366 2338 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.490877 2333 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:38.492306 2187 ts_tablet_manager.cc:1428] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.018s user 0.002s sys 0.004s
I20260927 19:37:38.484951 2338 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.498567 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.499703 2070 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 1: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:38.500658 2187 tablet_bootstrap.cc:492] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:38.507536 1891 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:38.508774 2187 tablet_bootstrap.cc:654] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.508545 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.510986 1962 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:38.511981 2244 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.521683 2185 ts_tablet_manager.cc:1397] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.169s user 0.026s sys 0.017s
I20260927 19:37:38.575356 2244 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.525537 2338 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.587812 2185 raft_consensus.cc:348] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.588769 2185 raft_consensus.cc:374] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.611277 2186 tablet_bootstrap.cc:492] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:38.618849 2186 ts_tablet_manager.cc:1397] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.238s user 0.032s sys 0.061s
I20260927 19:37:38.589089 2185 raft_consensus.cc:729] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.615839 2389 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.620328 2389 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.622319 2389 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.622087 2186 raft_consensus.cc:348] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.623548 2186 raft_consensus.cc:374] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.623876 2186 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.624153 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.624725 2186 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.625406 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.626741 2186 ts_tablet_manager.cc:1428] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 19:37:38.626022 2070 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:38.627784 2186 tablet_bootstrap.cc:492] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:38.633597 2186 tablet_bootstrap.cc:654] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.626937 2185 consensus_queue.cc:260] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.636224 1995 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:38.641963 2110 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.643719 2185 ts_tablet_manager.cc:1428] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.116s user 0.009s sys 0.000s
I20260927 19:37:38.643868 2389 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.644199 2389 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.644523 2389 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.644825 2185 tablet_bootstrap.cc:492] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.654054 2389 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.656683 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:38.657265 1995 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.658656 2185 tablet_bootstrap.cc:654] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.670543 2186 tablet_bootstrap.cc:492] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:38.670989 2186 ts_tablet_manager.cc:1397] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.004s
I20260927 19:37:38.670658 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.671623 2070 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.671978 1995 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:38.673399 2186 raft_consensus.cc:348] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.674494 2186 raft_consensus.cc:374] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.674726 2186 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.678619 2186 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.681200 2186 ts_tablet_manager.cc:1428] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260927 19:37:38.682029 2186 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:38.678503 2389 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.683418 2111 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.701655 2390 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.705003 2390 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.707041 2390 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.708496 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.708678 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:38.709918 2037 leader_election.cc:341] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:38.710577 2037 leader_election.cc:341] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.710611 2389 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:38.710983 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:38.712663 2389 raft_consensus.cc:686] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:38.713495 2186 tablet_bootstrap.cc:654] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.713631 2389 consensus_queue.cc:237] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.715590 2390 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.719208 2070 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:38.723657 2187 tablet_bootstrap.cc:492] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:38.724085 2187 ts_tablet_manager.cc:1397] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.224s user 0.012s sys 0.014s
I20260927 19:37:38.731009 2185 tablet_bootstrap.cc:492] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.731473 2185 ts_tablet_manager.cc:1397] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.011s
I20260927 19:37:38.734304 2187 raft_consensus.cc:348] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.735216 2187 raft_consensus.cc:374] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.735532 2187 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.739106 2187 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.740893 2185 raft_consensus.cc:348] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.741644 2185 raft_consensus.cc:374] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.741851 2187 ts_tablet_manager.cc:1428] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
I20260927 19:37:38.741921 2185 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.742880 2187 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:38.742883 2185 consensus_queue.cc:260] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.748848 2187 tablet_bootstrap.cc:654] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.749810 2185 ts_tablet_manager.cc:1428] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260927 19:37:38.750756 2185 tablet_bootstrap.cc:492] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.753126 2186 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:38.753809 2186 ts_tablet_manager.cc:1397] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.012s
I20260927 19:37:38.755990 2185 tablet_bootstrap.cc:654] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.755718 1890 catalog_manager.cc:5495] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:38.758623 1890 catalog_manager.cc:5495] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:38.769312 2185 tablet_bootstrap.cc:492] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.770082 2185 ts_tablet_manager.cc:1397] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20260927 19:37:38.773000 2390 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.773578 2390 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.775738 2390 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.777714 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.777741 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:38.779242 2037 leader_election.cc:341] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:38.779894 2037 leader_election.cc:341] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.780273 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:38.781307 2390 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.782064 2185 raft_consensus.cc:348] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.782580 2185 raft_consensus.cc:374] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.782800 2185 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.783345 2185 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.785480 2185 ts_tablet_manager.cc:1428] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20260927 19:37:38.785331 2186 raft_consensus.cc:348] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.786342 2186 raft_consensus.cc:374] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.786623 2186 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.786868 2185 tablet_bootstrap.cc:492] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.787194 2186 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.792544 2185 tablet_bootstrap.cc:654] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.801091 2185 tablet_bootstrap.cc:492] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.801554 2185 ts_tablet_manager.cc:1397] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260927 19:37:38.801481 2389 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.802021 2389 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.803921 2389 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.804008 2185 raft_consensus.cc:348] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.804719 2185 raft_consensus.cc:374] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.804970 2185 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.805572 2185 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.806077 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:38.807700 2111 leader_election.cc:341] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.807883 2185 ts_tablet_manager.cc:1428] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 19:37:38.808651 2185 tablet_bootstrap.cc:492] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.809774 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.810381 2070 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:38.813572 2186 ts_tablet_manager.cc:1428] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.059s user 0.007s sys 0.000s
I20260927 19:37:38.813863 2185 tablet_bootstrap.cc:654] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.814774 2186 tablet_bootstrap.cc:492] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:38.817252 2110 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:38.822196 2187 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:38.822274 2185 tablet_bootstrap.cc:492] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.822724 2187 ts_tablet_manager.cc:1397] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.080s user 0.016s sys 0.052s
I20260927 19:37:38.822762 2185 ts_tablet_manager.cc:1397] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.001s
I20260927 19:37:38.823228 2186 tablet_bootstrap.cc:654] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.823523 2389 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.823880 2389 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.824230 2389 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.825454 2187 raft_consensus.cc:348] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.826253 2187 raft_consensus.cc:374] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.825865 2185 raft_consensus.cc:348] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.826882 2185 raft_consensus.cc:374] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.827234 2185 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.826529 2187 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.828025 2187 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.828125 2185 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.830490 2187 ts_tablet_manager.cc:1428] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260927 19:37:38.830790 2185 ts_tablet_manager.cc:1428] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 19:37:38.831647 2185 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.832217 2187 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:38.836496 2244 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.836642 2333 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.837036 2244 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.837046 2333 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.837919 2185 tablet_bootstrap.cc:654] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.838873 2186 tablet_bootstrap.cc:492] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:38.839008 2333 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.839282 2186 ts_tablet_manager.cc:1397] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.000s
I20260927 19:37:38.839016 2244 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.841214 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.841814 2186 raft_consensus.cc:348] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.842581 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.842767 2186 raft_consensus.cc:374] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.843140 2186 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.843199 2142 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:38.841857 2143 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:38.843993 2186 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.844391 2387 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:38.846055 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.846932 1962 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.847731 2186 ts_tablet_manager.cc:1428] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260927 19:37:38.848379 2338 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.848780 2338 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.849099 2338 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.851032 2185 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.851052 2215 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.851593 2185 ts_tablet_manager.cc:1397] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20260927 19:37:38.851621 2215 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.853616 2215 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.854115 2185 raft_consensus.cc:348] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.854563 2333 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.854773 2185 raft_consensus.cc:374] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.854957 2333 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.855084 2185 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.855263 2186 tablet_bootstrap.cc:492] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:38.855368 2333 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.855718 2185 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.856890 2389 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.858114 2185 ts_tablet_manager.cc:1428] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260927 19:37:38.858968 2185 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.860772 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.861490 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:38.861652 2070 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
W20260927 19:37:38.862942 2111 leader_election.cc:341] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.863586 2388 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:38.865721 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:38.867075 2244 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.867128 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.867825 2069 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:38.867651 2244 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.871934 2185 tablet_bootstrap.cc:654] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.873294 2381 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.872330 2186 tablet_bootstrap.cc:654] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.873791 2381 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.876286 2381 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.876761 2381 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:38.877161 2381 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.877439 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:38.878896 1962 leader_election.cc:341] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.880932 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.892325 2338 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:38.894516 1962 leader_election.cc:341] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.894835 2338 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.896265 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:38.896849 2143 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.899313 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.899914 2070 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.901749 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.911396 2244 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.912338 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:38.913026 2142 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:38.914898 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.916137 2253 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:38.916500 2244 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:38.916618 2143 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:38.917032 2244 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:38.917512 2244 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.917763 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:38.879006 2381 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.918704 2381 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:38.919142 2381 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:38.919929 2381 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.920547 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.920406 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.921506 2069 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:38.924314 2389 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:38.924198 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:38.925011 2069 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.931610 2070 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:38.932078 2186 tablet_bootstrap.cc:492] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:38.932395 2185 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:38.932570 2186 ts_tablet_manager.cc:1397] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.077s user 0.019s sys 0.013s
I20260927 19:37:38.932912 2185 ts_tablet_manager.cc:1397] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.074s user 0.015s sys 0.022s
I20260927 19:37:38.933521 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:38.934317 2187 tablet_bootstrap.cc:654] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.934216 2071 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:38.935051 2186 raft_consensus.cc:348] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:38.936156 2037 leader_election.cc:341] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.933954 2333 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.936175 2186 raft_consensus.cc:374] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260927 19:37:38.942410 2037 leader_election.cc:341] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.943077 2186 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.942355 2185 raft_consensus.cc:348] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.943753 2185 raft_consensus.cc:374] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:38.944234 2185 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:38.943252 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:38.943737 2186 consensus_queue.cc:260] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.945050 2185 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.949659 2185 ts_tablet_manager.cc:1428] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.016s user 0.004s sys 0.003s
I20260927 19:37:38.950881 2185 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:38.946301 2390 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.955000 2186 ts_tablet_manager.cc:1428] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20260927 19:37:38.952525 2110 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:38.956612 2186 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
W20260927 19:37:38.953145 1961 leader_election.cc:341] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:38.962533 2389 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:38.965935 2389 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:38.966307 2338 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:38.966571 2244 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.966774 2389 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:38.969856 2333 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:38.987215 2186 tablet_bootstrap.cc:654] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:38.970666 1961 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:38.991043 2253 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:38.994516 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:38.995149 2142 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.994557 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:39.004491 2185 tablet_bootstrap.cc:654] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:39.005195 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52990.8us
I20260927 19:37:39.004040 2145 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.008952 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.009716 2070 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:38.991526 2244 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.002986 2338 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:38.988974 2069 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.022459 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.023448 2067 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.020910 2398 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.024371 2398 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.027107 2398 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.033809 2187 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:39.034477 2187 ts_tablet_manager.cc:1397] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.202s user 0.013s sys 0.002s
I20260927 19:37:39.042521 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:39.032843 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.042430 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:39.044122 2037 leader_election.cc:341] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:39.045079 2037 leader_election.cc:341] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.045733 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.046540 2390 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:39.047320 2145 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.051322 2338 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.051777 2338 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.052109 2338 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.055284 2070 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.060460 2338 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.032045 2142 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.064592 2338 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.065344 1962 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.081058 2338 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.081598 2338 raft_consensus.cc:686] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.084553 2338 consensus_queue.cc:237] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.085987 2067 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.086200 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.087299 2068 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.096381 2068 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.100082 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.100525 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:39.101245 2338 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.102344 2338 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.103189 2338 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.104902 2185 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:39.105340 2185 ts_tablet_manager.cc:1397] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.008s
I20260927 19:37:39.107875 2187 raft_consensus.cc:348] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.108273 2185 raft_consensus.cc:348] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:39.109566 1962 leader_election.cc:341] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.114166 2185 raft_consensus.cc:374] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.096884 1891 catalog_manager.cc:5495] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.114476 2185 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.115057 2185 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.114192 2187 raft_consensus.cc:374] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.117005 2187 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.113569 2398 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.118861 2398 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.120560 2185 ts_tablet_manager.cc:1428] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.015s user 0.004s sys 0.002s
I20260927 19:37:39.121906 2185 tablet_bootstrap.cc:492] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:39.117908 2187 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.120725 1892 catalog_manager.cc:5495] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:39.130590 2398 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.133688 2187 ts_tablet_manager.cc:1428] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.099s user 0.006s sys 0.003s
I20260927 19:37:39.133981 2333 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.134639 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.135818 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:39.138461 2185 tablet_bootstrap.cc:654] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.139285 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.141227 2187 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:39.146639 2187 tablet_bootstrap.cc:654] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.148602 2186 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:39.149154 2186 ts_tablet_manager.cc:1397] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.193s user 0.017s sys 0.005s
W20260927 19:37:39.150561 2037 leader_election.cc:341] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:39.151175 2037 leader_election.cc:341] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.151510 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.151751 2185 tablet_bootstrap.cc:492] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:39.152232 2185 ts_tablet_manager.cc:1397] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20260927 19:37:39.152213 2186 raft_consensus.cc:348] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.152676 2390 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:39.153802 2333 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.154289 2333 raft_consensus.cc:686] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.154877 2185 raft_consensus.cc:348] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.155594 2185 raft_consensus.cc:374] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.155119 2333 consensus_queue.cc:237] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.155866 2185 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.156490 2185 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.161741 2185 ts_tablet_manager.cc:1428] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 19:37:39.162632 2185 tablet_bootstrap.cc:492] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:39.168092 2187 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:39.168627 2185 tablet_bootstrap.cc:654] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.168678 2187 ts_tablet_manager.cc:1397] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.004s
I20260927 19:37:39.170053 2186 raft_consensus.cc:374] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.170472 2186 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.174053 2186 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.175561 2187 raft_consensus.cc:348] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.176712 2187 raft_consensus.cc:374] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.180274 2187 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.180840 2387 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.181020 2187 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.179981 2186 ts_tablet_manager.cc:1428] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.030s user 0.004s sys 0.001s
I20260927 19:37:39.182600 2186 tablet_bootstrap.cc:492] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:39.188932 2187 ts_tablet_manager.cc:1428] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.020s user 0.002s sys 0.003s
I20260927 19:37:39.191372 2187 tablet_bootstrap.cc:492] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:39.200621 2186 tablet_bootstrap.cc:654] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.200085 1891 catalog_manager.cc:5495] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.202184 1891 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.203955 1891 catalog_manager.cc:5495] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.210726 2187 tablet_bootstrap.cc:654] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.237157 2388 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:39.251549 2187 tablet_bootstrap.cc:492] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:39.252240 2187 ts_tablet_manager.cc:1397] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.004s
I20260927 19:37:39.253981 2333 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.254678 2333 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.260862 2333 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.262320 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.263005 2068 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.263695 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.264308 2142 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.267045 1961 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:39.263732 2186 tablet_bootstrap.cc:492] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:39.268673 2186 ts_tablet_manager.cc:1397] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.011s
I20260927 19:37:39.267988 2338 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.269097 2338 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.269697 2338 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.272543 2185 tablet_bootstrap.cc:492] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:39.278159 2185 ts_tablet_manager.cc:1397] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.116s user 0.033s sys 0.039s
I20260927 19:37:39.280861 2185 raft_consensus.cc:348] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.281359 2185 raft_consensus.cc:374] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.281639 2185 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.282167 2185 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.283934 2185 ts_tablet_manager.cc:1428] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260927 19:37:39.273902 2187 raft_consensus.cc:348] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.284296 2187 raft_consensus.cc:374] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.284554 2187 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.276441 2186 raft_consensus.cc:348] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.285185 2186 raft_consensus.cc:374] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.285472 2186 raft_consensus.cc:729] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.285539 2185 tablet_bootstrap.cc:492] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:39.286177 2187 consensus_queue.cc:260] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.287199 2186 consensus_queue.cc:260] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.288535 2186 ts_tablet_manager.cc:1428] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260927 19:37:39.289556 2186 tablet_bootstrap.cc:492] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:39.292508 2185 tablet_bootstrap.cc:654] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.296069 2187 ts_tablet_manager.cc:1428] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.043s user 0.006s sys 0.001s
I20260927 19:37:39.297403 2187 tablet_bootstrap.cc:492] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:39.302623 2187 tablet_bootstrap.cc:654] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.292129 2338 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.304107 2185 tablet_bootstrap.cc:492] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:39.304555 2185 ts_tablet_manager.cc:1397] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.014s
I20260927 19:37:39.306483 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:39.305886 2338 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.305691 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.307006 2185 raft_consensus.cc:348] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.309755 2142 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.310873 2185 raft_consensus.cc:374] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.311177 2185 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.311760 2185 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.310447 2067 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.313925 2185 ts_tablet_manager.cc:1428] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260927 19:37:39.314798 2185 tablet_bootstrap.cc:492] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:39.316198 2186 tablet_bootstrap.cc:654] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.317327 2142 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.328258 2390 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.330030 2390 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.332257 2390 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.333015 2388 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.333612 2388 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.334631 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:39.335278 1994 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:39.335772 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.336025 2388 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:39.336454 2144 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:39.338335 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:39.340556 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:39.341866 2338 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.342023 1994 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:39.342411 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.342556 2338 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.343448 2111 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.343918 2390 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.345618 2071 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:39.348975 2390 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.349521 2390 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.353595 2388 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.354246 2388 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.354630 2388 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.363562 2338 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.382047 2185 tablet_bootstrap.cc:654] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.385707 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.386546 2071 raft_consensus.cc:2379] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 1: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:39.387674 1962 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.380348 2067 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.388837 2333 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.389914 2338 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.388612 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.390515 2338 raft_consensus.cc:686] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.390278 2333 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.391410 2338 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.391814 1962 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.392232 2333 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.392899 2333 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.394369 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.394624 2388 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:39.396052 1962 leader_election.cc:341] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.397019 2388 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:39.398720 2187 tablet_bootstrap.cc:492] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:39.399111 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.399330 2187 ts_tablet_manager.cc:1397] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.004s
I20260927 19:37:39.399036 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:39.400141 1994 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.402714 2187 raft_consensus.cc:348] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.403642 2187 raft_consensus.cc:374] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.404016 2187 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.404835 2187 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.409049 2215 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.409564 2215 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.411479 2215 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.411864 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.417838 2067 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.418165 1994 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:39.419104 2333 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:39.425190 2185 tablet_bootstrap.cc:492] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:39.426426 2186 tablet_bootstrap.cc:492] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:39.425716 2185 ts_tablet_manager.cc:1397] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.009s
I20260927 19:37:39.426956 2186 ts_tablet_manager.cc:1397] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.138s user 0.017s sys 0.017s
I20260927 19:37:39.411096 2390 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.419485 2111 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.421319 2426 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.428078 2071 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:37:39.428341 2426 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.429535 2389 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.430086 2389 raft_consensus.cc:686] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:39.429040 2185 raft_consensus.cc:348] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.430904 2389 consensus_queue.cc:237] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.431444 2185 raft_consensus.cc:374] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.431702 2185 raft_consensus.cc:729] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.432274 2185 consensus_queue.cc:260] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.421650 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:39.421893 2187 ts_tablet_manager.cc:1428] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.022s user 0.010s sys 0.000s
I20260927 19:37:39.422519 2388 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:37:39.435211 2187 tablet_bootstrap.cc:492] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:39.422819 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.435961 2426 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.422997 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:37:39.436658 2037 leader_election.cc:341] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.437107 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:39.437556 2037 leader_election.cc:341] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.437916 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.438097 2388 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.438845 2215 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:39.437934 2338 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.439435 2338 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.439785 2338 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.439532 1890 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.443099 2187 tablet_bootstrap.cc:654] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.445642 2186 raft_consensus.cc:348] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.446364 2186 raft_consensus.cc:374] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.446604 2186 raft_consensus.cc:729] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.447204 2186 consensus_queue.cc:260] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.449664 2186 ts_tablet_manager.cc:1428] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260927 19:37:39.450030 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.450616 2186 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:39.451355 2338 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.456928 2186 tablet_bootstrap.cc:654] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.457778 1961 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.470055 2185 ts_tablet_manager.cc:1428] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260927 19:37:39.471074 2185 tablet_bootstrap.cc:492] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:39.472100 2426 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:37:39.472831 2426 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.473258 2426 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.475770 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:37:39.477557 1962 leader_election.cc:341] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.478340 2426 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.479475 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:39.480013 2390 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.480204 1996 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 1: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:39.481092 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:39.490484 2333 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.491062 1891 catalog_manager.cc:5495] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:39.493681 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.494098 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.494292 2145 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.497462 2425 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 19:37:39.499560 1962 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.500415 2381 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.500783 2381 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.501194 2381 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.484464 2338 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.525329 2338 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.526047 2338 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.530439 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.531032 2145 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.532912 1962 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.533932 2381 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.535202 2333 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:39.535583 2333 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:39.529917 2338 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.535889 2333 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.543560 2187 tablet_bootstrap.cc:492] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:39.544203 2187 ts_tablet_manager.cc:1397] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.109s user 0.014s sys 0.011s
I20260927 19:37:39.533245 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.547291 2067 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.545171 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.549980 2068 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.553213 2067 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.554740 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:39.555677 2069 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:39.579782 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.580514 2244 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.581040 2244 raft_consensus.cc:686] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.583525 2185 tablet_bootstrap.cc:654] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.581910 2244 consensus_queue.cc:237] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.584116 2333 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.621531 2422 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:39.631860 2390 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.632457 2390 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.634624 2186 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:39.635211 2186 ts_tablet_manager.cc:1397] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.185s user 0.035s sys 0.064s
I20260927 19:37:39.635622 2390 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.638635 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:39.640743 2037 leader_election.cc:341] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.644472 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.645609 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:39.646013 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.646394 2142 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.646574 2066 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.648918 2186 raft_consensus.cc:348] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.650048 2186 raft_consensus.cc:374] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.650334 2186 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.650982 2186 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.654529 2186 ts_tablet_manager.cc:1428] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.019s user 0.004s sys 0.003s
I20260927 19:37:39.632480 2333 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.656207 2186 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
W20260927 19:37:39.659535 2037 leader_election.cc:341] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.660178 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.644944 2381 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.661725 2390 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:39.668458 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:39.669060 2145 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:39.678676 2145 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.681051 2187 raft_consensus.cc:348] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.681797 2187 raft_consensus.cc:374] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.682041 2187 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.682657 2187 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.693931 2333 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.699013 2066 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.706732 2187 ts_tablet_manager.cc:1428] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.162s user 0.006s sys 0.000s
I20260927 19:37:39.707674 2187 tablet_bootstrap.cc:492] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:39.715986 1962 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.718380 2244 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.719069 2142 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:39.719913 2333 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.720384 2333 raft_consensus.cc:686] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.722496 2333 consensus_queue.cc:237] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.727612 1892 catalog_manager.cc:5495] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.730535 2390 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:37:39.733278 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:39.752166 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:39.797537 2458 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.798281 2458 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.801064 2458 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.762089 2390 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.809499 2066 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:39.821936 1962 consensus_peers.cc:487] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Couldn't send request to peer 69f09866d0a1477585f05b2f82ff4c67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:39.822698 2422 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:39.824887 1962 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:39.826910 2333 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:37:39.827226 2185 tablet_bootstrap.cc:492] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:39.828516 2185 ts_tablet_manager.cc:1397] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.358s user 0.029s sys 0.053s
I20260927 19:37:39.829104 2390 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:39.827195 2244 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:39.836257 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.839354 2244 raft_consensus.cc:686] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:39.840690 2244 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.844223 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:39.844856 1993 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:37:39.849651 2462 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260927 19:37:39.858201 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:39.859059 2144 raft_consensus.cc:2379] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:39.859445 2187 tablet_bootstrap.cc:654] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:39.862736 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:39.866365 2215 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:37:39.866902 2215 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:37:39.867362 2215 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:37:39.873977 2185 raft_consensus.cc:348] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.874543 2185 raft_consensus.cc:374] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:39.874791 2185 raft_consensus.cc:729] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:39.875293 2185 consensus_queue.cc:260] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.883783 2185 ts_tablet_manager.cc:1428] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.055s user 0.001s sys 0.005s
I20260927 19:37:39.884845 2185 tablet_bootstrap.cc:492] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:39.885224 1892 catalog_manager.cc:5495] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:39.885922 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:39.888409 1961 leader_election.cc:341] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:39.881230 2387 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:39.899201 2387 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.932224 2387 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:39.996359 2215 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:39.976681 2066 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.008658 2186 tablet_bootstrap.cc:654] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.011917 2458 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.012432 2458 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.014420 2458 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.024087 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:40.028167 2111 leader_election.cc:341] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.029490 2253 consensus_queue.cc:1035] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260927 19:37:40.035216 1962 leader_election.cc:341] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.035684 1962 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.037870 2185 tablet_bootstrap.cc:654] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.041105 2186 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.041313 2187 tablet_bootstrap.cc:492] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.041797 2186 ts_tablet_manager.cc:1397] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.386s user 0.027s sys 0.056s
I20260927 19:37:40.041888 2187 ts_tablet_manager.cc:1397] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.334s user 0.024s sys 0.053s
I20260927 19:37:40.024045 2215 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.044302 2186 raft_consensus.cc:348] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.045076 2186 raft_consensus.cc:374] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.045375 2186 raft_consensus.cc:729] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.045989 2186 consensus_queue.cc:260] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.048537 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.049185 2458 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.049345 2187 raft_consensus.cc:348] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.050139 2187 raft_consensus.cc:374] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.050401 2187 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.051059 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:40.052516 2143 raft_consensus.cc:2379] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:40.053398 2186 ts_tablet_manager.cc:1428] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260927 19:37:40.054311 2186 tablet_bootstrap.cc:492] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:40.050967 2187 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.051474 2338 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:40.057837 2187 ts_tablet_manager.cc:1428] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260927 19:37:40.060720 2215 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.067039 2215 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.067584 2187 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.068949 2215 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.069641 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:40.074839 2388 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.075387 2388 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.076386 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.077050 2067 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:40.078192 2388 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.082057 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.062520 2186 tablet_bootstrap.cc:654] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.065032 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.083485 2187 tablet_bootstrap.cc:654] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.083791 2068 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
W20260927 19:37:40.083298 1962 consensus_peers.cc:487] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Couldn't send request to peer 69f09866d0a1477585f05b2f82ff4c67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260927 19:37:40.086490 1962 leader_election.cc:341] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.092712 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.097892 2110 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:40.100412 2388 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.100821 2388 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.101161 2388 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.101377 2253 consensus_queue.cc:1035] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.105085 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:40.106563 2037 leader_election.cc:341] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.107952 2338 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.108596 2185 tablet_bootstrap.cc:492] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:40.109087 2185 ts_tablet_manager.cc:1397] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.224s user 0.015s sys 0.000s
I20260927 19:37:40.102710 1892 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:40.111940 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.114245 2381 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.114611 2381 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.114959 2381 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.124017 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.130698 2462 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.131037 2141 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.135000 2381 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.141119 2111 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:40.142824 2185 raft_consensus.cc:348] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.143899 2185 raft_consensus.cc:374] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.144240 2185 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.155817 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.156486 2063 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.163650 2381 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.164587 2063 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.164780 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:37:40.166134 1962 leader_election.cc:341] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.145090 2185 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.171406 1993 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:37:40.173159 2388 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.176595 2185 ts_tablet_manager.cc:1428] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20260927 19:37:40.176630 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:40.177393 1996 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.179200 2187 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.179674 2187 ts_tablet_manager.cc:1397] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.112s user 0.032s sys 0.019s
I20260927 19:37:40.180994 2185 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:40.184172 1996 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:40.184303 2187 raft_consensus.cc:348] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.185446 2187 raft_consensus.cc:374] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.185953 2187 raft_consensus.cc:729] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.186986 2111 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:40.186765 2187 consensus_queue.cc:260] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.187929 2388 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.188714 2388 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.189206 2388 raft_consensus.cc:686] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:37:40.191279 2037 leader_election.cc:341] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.194475 2037 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.195397 2469 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.196133 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.196892 2469 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:37:40.192210 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.197347 2469 raft_consensus.cc:686] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:40.197497 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.197644 2063 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.198177 2469 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.198945 2422 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.199887 2422 raft_consensus.cc:686] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:40.200757 2422 consensus_queue.cc:237] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.202544 2338 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.207391 2186 tablet_bootstrap.cc:492] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.208071 2186 ts_tablet_manager.cc:1397] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.006s
I20260927 19:37:40.211192 2186 raft_consensus.cc:348] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.211977 2186 raft_consensus.cc:374] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.212287 2186 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.213248 2187 ts_tablet_manager.cc:1428] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.033s user 0.008s sys 0.003s
I20260927 19:37:40.214591 2187 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.212950 2186 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.220317 2186 ts_tablet_manager.cc:1428] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260927 19:37:40.221313 2186 tablet_bootstrap.cc:492] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:40.225335 2390 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.225943 2390 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.227769 2390 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.190184 2388 consensus_queue.cc:237] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.230902 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.231216 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.232946 2458 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 19:37:40.235194 2037 leader_election.cc:341] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:40.235823 2037 leader_election.cc:341] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.236167 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.236837 2390 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.246052 2185 tablet_bootstrap.cc:654] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.246382 2387 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.247383 2387 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.246114 2389 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.255823 2387 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.246829 2187 tablet_bootstrap.cc:654] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.262799 2186 tablet_bootstrap.cc:654] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.272619 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.273693 1892 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:40.257005 2063 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
W20260927 19:37:40.283962 2110 leader_election.cc:341] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.286278 2338 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.267168 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:40.294268 2111 leader_election.cc:341] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.294816 2111 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:40.294629 2422 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260927 19:37:40.299105 2389 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.301853 1891 catalog_manager.cc:5495] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:40.352198 2187 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.352769 2187 ts_tablet_manager.cc:1397] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.139s user 0.016s sys 0.024s
I20260927 19:37:40.354797 2185 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:40.356544 2185 ts_tablet_manager.cc:1397] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.176s user 0.013s sys 0.012s
I20260927 19:37:40.360103 2187 raft_consensus.cc:348] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.362566 2187 raft_consensus.cc:374] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.362931 2187 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.361831 2185 raft_consensus.cc:348] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.364050 2185 raft_consensus.cc:374] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.365801 2185 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.364741 2187 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.366865 2185 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.375398 2185 ts_tablet_manager.cc:1428] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260927 19:37:40.377628 2185 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:40.376367 2187 ts_tablet_manager.cc:1428] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260927 19:37:40.379698 2187 tablet_bootstrap.cc:492] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.386082 2187 tablet_bootstrap.cc:654] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.394143 2185 tablet_bootstrap.cc:654] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.400903 2185 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:40.401348 2185 ts_tablet_manager.cc:1397] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.003s
I20260927 19:37:40.402113 1892 catalog_manager.cc:5495] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:40.404809 2185 raft_consensus.cc:348] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.405669 2185 raft_consensus.cc:374] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.405933 2185 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.407779 2187 tablet_bootstrap.cc:492] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.408222 2187 ts_tablet_manager.cc:1397] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20260927 19:37:40.410665 2187 raft_consensus.cc:348] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.411325 2187 raft_consensus.cc:374] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.411571 2187 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.412220 2187 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.414764 2187 ts_tablet_manager.cc:1428] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 19:37:40.415580 2187 tablet_bootstrap.cc:492] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.417855 2185 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.418395 2186 tablet_bootstrap.cc:492] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.418852 2186 ts_tablet_manager.cc:1397] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.198s user 0.022s sys 0.035s
I20260927 19:37:40.419782 2185 ts_tablet_manager.cc:1428] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260927 19:37:40.420490 2185 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:40.421617 2186 raft_consensus.cc:348] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.422286 2186 raft_consensus.cc:374] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.422536 2186 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.423132 2186 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.424726 2186 ts_tablet_manager.cc:1428] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 19:37:40.426529 2185 tablet_bootstrap.cc:654] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.427323 2187 tablet_bootstrap.cc:654] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.429499 2186 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:40.435465 2186 tablet_bootstrap.cc:654] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.444911 2185 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:40.445385 2185 ts_tablet_manager.cc:1397] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.004s
I20260927 19:37:40.454756 2338 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.455333 2338 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.456487 2185 raft_consensus.cc:348] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.457064 2338 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.458773 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.459424 2071 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:40.460353 2140 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.460350 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.461076 2141 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:40.457226 2185 raft_consensus.cc:374] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.461925 2185 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.462584 2185 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.465241 2187 tablet_bootstrap.cc:492] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.465677 2185 ts_tablet_manager.cc:1428] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260927 19:37:40.465780 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:40.465709 2187 ts_tablet_manager.cc:1397] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.050s user 0.016s sys 0.010s
I20260927 19:37:40.466918 2422 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.467304 2422 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.467623 2422 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.468780 2187 raft_consensus.cc:348] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.460369 2186 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.469527 2187 raft_consensus.cc:374] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.469861 2186 ts_tablet_manager.cc:1397] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.041s user 0.016s sys 0.000s
I20260927 19:37:40.470094 2185 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:40.469810 2187 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.473076 2186 raft_consensus.cc:348] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.473830 2186 raft_consensus.cc:374] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.474097 2186 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.474798 2186 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.475226 2389 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.475821 2389 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.478120 2186 ts_tablet_manager.cc:1428] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260927 19:37:40.479126 2186 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:40.479444 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.479429 2422 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.484695 2186 tablet_bootstrap.cc:654] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.481899 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.480849 2389 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.508104 2422 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.483201 2187 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.509673 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:40.510421 2141 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.510818 2187 ts_tablet_manager.cc:1428] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.044s user 0.004s sys 0.000s
I20260927 19:37:40.511538 2187 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.515103 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.516644 2185 tablet_bootstrap.cc:654] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.516414 2071 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.517287 2187 tablet_bootstrap.cc:654] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.480245 1996 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:37:40.529978 2141 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.530251 2111 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:37:40.531399 2389 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:37:40.532977 1962 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:40.533706 2422 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.531767 2389 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.534338 2422 raft_consensus.cc:686] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:40.534507 2389 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:40.535434 2422 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.553980 2071 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.558575 2388 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.559361 2388 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.556970 2389 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.571650 2186 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.572464 2186 ts_tablet_manager.cc:1397] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.094s user 0.020s sys 0.044s
I20260927 19:37:40.578684 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.579627 2187 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.580127 2187 ts_tablet_manager.cc:1397] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.003s
I20260927 19:37:40.582612 2389 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.582774 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.583348 1891 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
W20260927 19:37:40.592121 2111 leader_election.cc:341] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.593129 2187 raft_consensus.cc:348] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.594282 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.594971 2071 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:40.588821 2388 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.593873 2187 raft_consensus.cc:374] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.596108 2187 raft_consensus.cc:729] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.595990 2110 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:40.598016 2388 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.598977 2388 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.599323 2388 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.596920 2187 consensus_queue.cc:260] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.600040 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:40.600605 1996 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:40.601696 2187 ts_tablet_manager.cc:1428] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260927 19:37:40.602626 2187 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.602967 2186 raft_consensus.cc:348] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.603564 2186 raft_consensus.cc:374] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.603794 2186 raft_consensus.cc:729] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.607390 2186 consensus_queue.cc:260] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.611351 2185 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:40.611864 2185 ts_tablet_manager.cc:1397] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.142s user 0.031s sys 0.006s
I20260927 19:37:40.612058 1996 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:37:40.617285 2185 raft_consensus.cc:348] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.618043 2185 raft_consensus.cc:374] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.618305 2185 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.619463 2186 ts_tablet_manager.cc:1428] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20260927 19:37:40.618896 2185 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.620116 2111 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:37:40.620677 2186 tablet_bootstrap.cc:492] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:40.621397 2185 ts_tablet_manager.cc:1428] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260927 19:37:40.627084 2185 tablet_bootstrap.cc:492] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:40.622205 2389 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:37:40.627739 2389 raft_consensus.cc:686] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:40.628525 2389 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.629546 2186 tablet_bootstrap.cc:654] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.630427 2388 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.632405 2388 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.633139 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:40.633496 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.634122 2071 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:40.634965 2111 leader_election.cc:341] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.635120 2187 tablet_bootstrap.cc:654] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.642621 2185 tablet_bootstrap.cc:654] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.642688 2071 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:40.645500 2110 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:40.646546 2388 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.648186 2388 raft_consensus.cc:686] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:40.649174 2388 consensus_queue.cc:237] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.663020 2422 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.663635 2422 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.668953 1996 raft_consensus.cc:1226] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:37:40.670488 2390 consensus_queue.cc:1046] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:40.671273 2469 raft_consensus.cc:3032] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:37:40.671598 2469 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:40.672339 2469 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.673516 2469 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:37:40.681622 2187 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.682312 2187 ts_tablet_manager.cc:1397] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.080s user 0.019s sys 0.003s
I20260927 19:37:40.689692 2187 raft_consensus.cc:348] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.690593 2187 raft_consensus.cc:374] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.690960 2187 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.690590 2462 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.691928 2187 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.694237 2462 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.695590 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.696156 2462 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.696357 2063 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:40.696815 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.697460 2067 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:40.698108 2187 ts_tablet_manager.cc:1428] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260927 19:37:40.699085 1961 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:40.698240 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.699473 2187 tablet_bootstrap.cc:492] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.700934 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:40.702234 2338 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.702728 2338 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.703208 2338 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.704470 2381 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.704877 2381 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.705392 2381 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.715384 2185 tablet_bootstrap.cc:492] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:40.716055 2185 ts_tablet_manager.cc:1397] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.089s user 0.014s sys 0.006s
I20260927 19:37:40.717252 2422 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:37:40.718395 1962 leader_election.cc:341] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.728786 2338 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.733014 2338 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.734452 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.735081 2071 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.736114 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.736951 2141 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:40.738451 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:37:40.743085 1962 consensus_peers.cc:487] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Couldn't send request to peer 69f09866d0a1477585f05b2f82ff4c67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260927 19:37:40.743711 1962 leader_election.cc:341] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.748924 2388 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.748760 2185 raft_consensus.cc:348] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.749672 2185 raft_consensus.cc:374] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.749950 2185 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.750803 2185 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.752000 2187 tablet_bootstrap.cc:654] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.752882 2185 ts_tablet_manager.cc:1428] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260927 19:37:40.741674 2071 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.754225 2185 tablet_bootstrap.cc:492] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:40.741771 2381 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.754877 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.755847 2458 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.756244 2186 tablet_bootstrap.cc:492] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.756376 2381 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.756721 2186 ts_tablet_manager.cc:1397] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.136s user 0.027s sys 0.027s
I20260927 19:37:40.758777 1892 catalog_manager.cc:5495] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:40.759862 1892 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 3, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:40.756285 2458 raft_consensus.cc:686] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:40.765218 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.765851 2067 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.766949 2338 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.767443 2338 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.774992 2390 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.775507 2390 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.777504 2390 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.778836 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.778612 2458 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:40.780033 2037 leader_election.cc:341] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.782773 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.782963 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:40.771348 2186 raft_consensus.cc:348] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.783810 2186 raft_consensus.cc:374] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.784129 2186 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.783510 2145 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.784889 2186 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:40.785912 2037 leader_election.cc:341] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.786639 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.787182 2390 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.789408 2338 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.790850 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.791936 2253 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.792385 2253 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.792604 2145 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.794811 2253 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.795100 1962 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:40.795781 2338 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.796245 2338 raft_consensus.cc:686] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:40.797190 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.798033 2071 raft_consensus.cc:2379] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 1: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:37:40.797812 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:40.799394 2111 leader_election.cc:341] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.799242 2422 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:40.801971 2186 ts_tablet_manager.cc:1428] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.045s user 0.004s sys 0.004s
I20260927 19:37:40.801282 2338 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.802552 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.804134 1961 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.805744 2186 tablet_bootstrap.cc:492] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:40.805783 2422 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.805791 2067 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:40.806003 2253 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:40.807495 2066 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:40.808058 2390 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.824437 2187 tablet_bootstrap.cc:492] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:40.824506 2186 tablet_bootstrap.cc:654] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.825381 2187 ts_tablet_manager.cc:1397] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.006s
I20260927 19:37:40.828855 2187 raft_consensus.cc:348] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.830911 2187 raft_consensus.cc:374] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:40.831192 2187 raft_consensus.cc:729] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:40.832056 2187 consensus_queue.cc:260] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.833530 2469 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.820333 2390 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.834144 2469 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.834944 2187 ts_tablet_manager.cc:1428] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260927 19:37:40.835601 2390 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.836198 2469 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:40.836797 2110 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:40.837705 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:40.838218 2389 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.838589 2389 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.838935 2389 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.838871 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:40.841333 2037 leader_election.cc:341] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:40.841970 2037 leader_election.cc:341] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.848237 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:40.849745 2037 leader_election.cc:341] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.850391 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:40.859501 2422 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.860710 2187 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:40.864743 2390 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:40.889477 2389 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.892201 2389 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.895741 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:37:40.901885 2111 leader_election.cc:341] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:40.906564 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:40.907302 2067 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.909636 2185 tablet_bootstrap.cc:654] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.922323 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:40.923278 2140 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:40.924502 2387 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:40.925053 2387 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.927947 2387 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:40.930012 2067 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:40.938089 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:40.939024 2110 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:40.940178 2390 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:40.940567 2390 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:40.940900 2390 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:40.950690 2253 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:40.951247 2253 raft_consensus.cc:686] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:40.952070 2253 consensus_queue.cc:237] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:40.961136 1892 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:40.962872 1892 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:40.974992 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:40.975805 2067 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:40.976074 2187 tablet_bootstrap.cc:654] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:40.976830 2186 tablet_bootstrap.cc:492] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:40.977417 2186 ts_tablet_manager.cc:1397] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.172s user 0.022s sys 0.038s
W20260927 19:37:40.983587 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:41.021446 2110 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:37:41.051321 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 140819us
I20260927 19:37:41.055413 2381 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:41.055859 2422 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:41.055716 2186 raft_consensus.cc:348] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.056459 2186 raft_consensus.cc:374] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.056321 2422 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.056732 2186 raft_consensus.cc:729] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.057451 2186 consensus_queue.cc:260] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.024399 2390 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.070350 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:41.066196 2253 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:41.071903 2253 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:41.058384 2422 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.072263 2253 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.069576 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:41.065454 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 80337.7us
I20260927 19:37:41.081924 2185 tablet_bootstrap.cc:492] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:41.082337 2185 ts_tablet_manager.cc:1397] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.328s user 0.022s sys 0.013s
W20260927 19:37:41.085009 2111 leader_election.cc:341] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:41.098915 1962 leader_election.cc:341] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.102620 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:37:41.108349 1961 leader_election.cc:341] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.108759 1961 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:41.107954 1893 catalog_manager.cc:5495] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:41.115088 2458 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:41.094396 2253 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.124065 2186 ts_tablet_manager.cc:1428] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.146s user 0.018s sys 0.017s
I20260927 19:37:41.127841 2186 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:41.105521 2517 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260927 19:37:41.142032 2185 raft_consensus.cc:348] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.142798 2185 raft_consensus.cc:374] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.143050 2185 raft_consensus.cc:729] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.143839 2185 consensus_queue.cc:260] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.147006 2185 ts_tablet_manager.cc:1428] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.064s user 0.006s sys 0.001s
W20260927 19:37:41.153268 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 62711.3us
I20260927 19:37:41.162410 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:41.155828 2390 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.253413 2186 tablet_bootstrap.cc:654] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.269922 2185 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:41.274755 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:41.275509 2143 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:41.277928 2037 leader_election.cc:341] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.315174 2187 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:41.316164 2187 ts_tablet_manager.cc:1397] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.468s user 0.018s sys 0.019s
I20260927 19:37:41.327032 2071 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260927 19:37:41.327454 2071 pending_rounds.cc:76] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 1
I20260927 19:37:41.327702 2071 pending_rounds.cc:97] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:37:41.330231 2071 raft_consensus.cc:2866] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:37:41.335991 2338 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:41.326654 2143 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:41.339766 2253 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:41.344101 2187 raft_consensus.cc:348] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.345083 2187 raft_consensus.cc:374] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.345327 2187 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.345933 2187 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.347536 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:41.348038 2187 ts_tablet_manager.cc:1428] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260927 19:37:41.348165 1996 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.348877 2187 tablet_bootstrap.cc:492] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:41.359542 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:41.363071 2522 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:41.363647 2522 raft_consensus.cc:686] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:41.364454 2522 consensus_queue.cc:237] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.350908 2524 consensus_queue.cc:1035] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260927 19:37:41.371717 2387 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:41.375149 2186 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:41.375783 2186 ts_tablet_manager.cc:1397] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.248s user 0.010s sys 0.006s
I20260927 19:37:41.378188 2186 raft_consensus.cc:348] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.378717 2186 raft_consensus.cc:374] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.382714 1996 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:41.383757 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:41.384443 2066 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.378937 2186 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.388819 2527 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:41.392381 2186 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.395439 2141 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.397997 2185 tablet_bootstrap.cc:654] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.400903 2186 ts_tablet_manager.cc:1428] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260927 19:37:41.401818 2111 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:41.403371 2388 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:41.401821 2186 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:41.403861 2388 raft_consensus.cc:686] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:41.414361 2388 consensus_queue.cc:237] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.420626 1994 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:37:41.404294 2527 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260927 19:37:41.427512 2187 tablet_bootstrap.cc:654] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.437220 2469 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:41.441272 2469 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.444928 2469 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.445459 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:41.449777 2387 consensus_queue.cc:1035] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:37:41.453235 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:41.458702 2037 leader_election.cc:341] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.453670 1994 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:41.468914 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:41.471879 1893 catalog_manager.cc:5495] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:41.466185 2066 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:41.477711 2521 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:41.475744 2469 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:41.483880 2469 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:41.484303 2469 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.499504 2186 tablet_bootstrap.cc:654] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.469386 2253 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260927 19:37:41.481496 2521 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.507601 2521 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.517191 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:41.541399 2037 leader_election.cc:341] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.548604 2185 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:41.549050 2185 ts_tablet_manager.cc:1397] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.279s user 0.013s sys 0.011s
W20260927 19:37:41.526834 2535 log.cc:924] Time spent T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260927 19:37:41.554872 1891 catalog_manager.cc:5495] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:41.549255 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:41.561668 1994 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:41.564265 2469 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.566794 2469 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.569895 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:41.574515 2518 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:37:41.619246 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:41.633468 1995 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.665755 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:41.689891 2523 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:41.697436 2187 tablet_bootstrap.cc:492] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:41.698135 2187 ts_tablet_manager.cc:1397] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.349s user 0.038s sys 0.039s
W20260927 19:37:41.695271 2037 leader_election.cc:341] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.700526 2185 raft_consensus.cc:348] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.703563 2185 raft_consensus.cc:374] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.704162 2185 raft_consensus.cc:729] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.706475 2185 consensus_queue.cc:260] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.698181 2523 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:41.715473 2523 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.721741 2187 raft_consensus.cc:348] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.722527 2187 raft_consensus.cc:374] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.722797 2187 raft_consensus.cc:729] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.723475 2187 consensus_queue.cc:260] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.729283 2186 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:41.729804 2186 ts_tablet_manager.cc:1397] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.328s user 0.051s sys 0.027s
I20260927 19:37:41.733970 1995 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:41.734917 2187 ts_tablet_manager.cc:1428] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260927 19:37:41.735774 2187 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:41.739377 2186 raft_consensus.cc:348] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.740094 2186 raft_consensus.cc:374] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.740363 2186 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.740929 2186 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.743105 2186 ts_tablet_manager.cc:1428] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260927 19:37:41.743954 2186 tablet_bootstrap.cc:492] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:41.744958 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:41.738062 2547 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260927 19:37:41.747274 2185 ts_tablet_manager.cc:1428] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.198s user 0.012s sys 0.006s
I20260927 19:37:41.749734 2185 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:41.753682 2469 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:41.754233 2469 raft_consensus.cc:686] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:41.755044 2469 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.754413 2523 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.761929 2523 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.765303 2186 tablet_bootstrap.cc:654] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.777494 1996 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.778580 2185 tablet_bootstrap.cc:654] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.809583 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:41.812196 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:41.812914 1993 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.820134 1993 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
W20260927 19:37:41.822139 2037 leader_election.cc:341] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.837512 2522 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260927 19:37:41.840392 2186 tablet_bootstrap.cc:492] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:41.840852 2186 ts_tablet_manager.cc:1397] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.010s
I20260927 19:37:41.848730 2187 tablet_bootstrap.cc:654] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.851442 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:37:41.858711 2037 consensus_peers.cc:487] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 -> Peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Couldn't send request to peer 3d1bc2cd9efe43399ddba396f5ddff85. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:41.859364 2186 raft_consensus.cc:348] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.859953 2186 raft_consensus.cc:374] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.860239 2186 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.856958 2522 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:41.862004 2547 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:41.864363 2186 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.857957 2143 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:41.864748 2523 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:41.869498 2522 raft_consensus.cc:686] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:41.873229 2523 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.870791 2522 consensus_queue.cc:237] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.875258 2523 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:41.875898 1893 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:41.882103 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:41.885214 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:41.888410 2037 leader_election.cc:341] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.898641 2186 ts_tablet_manager.cc:1428] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.058s user 0.006s sys 0.002s
I20260927 19:37:41.899910 2186 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
W20260927 19:37:41.901006 2037 leader_election.cc:341] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:41.905588 2037 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:41.912919 2390 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:41.947734 2547 consensus_queue.cc:1035] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:41.977859 2185 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:41.978348 2185 ts_tablet_manager.cc:1397] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.229s user 0.028s sys 0.019s
I20260927 19:37:41.979777 2186 tablet_bootstrap.cc:654] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:41.986941 2522 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:41.987543 2185 raft_consensus.cc:348] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.988545 2185 raft_consensus.cc:374] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:41.989018 2185 raft_consensus.cc:729] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:41.988585 2522 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:41.993273 1893 catalog_manager.cc:5495] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:41.995217 2185 consensus_queue.cc:260] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.007735 2185 ts_tablet_manager.cc:1428] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20260927 19:37:42.012634 2185 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:42.025878 2187 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:42.026404 2187 ts_tablet_manager.cc:1397] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.291s user 0.023s sys 0.020s
I20260927 19:37:42.032001 2522 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.035241 2187 raft_consensus.cc:348] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.036018 2187 raft_consensus.cc:374] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.036288 2187 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.031116 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:42.038509 2037 leader_election.cc:341] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.037056 2187 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.041126 2187 ts_tablet_manager.cc:1428] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.014s user 0.003s sys 0.002s
I20260927 19:37:42.043944 2187 tablet_bootstrap.cc:492] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:42.046619 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:37:42.051254 2037 leader_election.cc:341] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.051669 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:42.054796 2550 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:42.055866 2547 consensus_queue.cc:1035] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:42.109457 2187 tablet_bootstrap.cc:654] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.114178 2185 tablet_bootstrap.cc:654] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:42.115705 2560 log.cc:924] Time spent T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260927 19:37:42.118129 2186 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:42.118623 2186 ts_tablet_manager.cc:1397] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.219s user 0.030s sys 0.038s
I20260927 19:37:42.121896 2186 raft_consensus.cc:348] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.123370 2186 raft_consensus.cc:374] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.123683 2186 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.124259 2186 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.126061 2187 tablet_bootstrap.cc:492] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:42.126537 2187 ts_tablet_manager.cc:1397] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.083s user 0.015s sys 0.000s
I20260927 19:37:42.126827 2186 ts_tablet_manager.cc:1428] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 19:37:42.127785 2186 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:42.129540 2187 raft_consensus.cc:348] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.130431 2187 raft_consensus.cc:374] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.130821 2187 raft_consensus.cc:729] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
W20260927 19:37:42.167011 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 46515.8us
W20260927 19:37:42.193053 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:42.194115 2185 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:42.195272 2185 ts_tablet_manager.cc:1397] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.184s user 0.011s sys 0.007s
I20260927 19:37:42.209609 2186 tablet_bootstrap.cc:654] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.131528 2187 consensus_queue.cc:260] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.220499 2550 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:42.225394 2185 raft_consensus.cc:348] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.236754 2185 raft_consensus.cc:374] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.235535 2187 ts_tablet_manager.cc:1428] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.108s user 0.003s sys 0.005s
I20260927 19:37:42.237105 2185 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.238376 2185 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.242704 2185 ts_tablet_manager.cc:1428] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.047s user 0.008s sys 0.004s
I20260927 19:37:42.244217 2185 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:42.248750 2187 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:42.257741 2187 tablet_bootstrap.cc:654] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.262072 2186 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:42.263074 2186 ts_tablet_manager.cc:1397] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.013s
I20260927 19:37:42.306860 2139 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.307792 2390 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:42.314978 2185 tablet_bootstrap.cc:654] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.313782 2186 raft_consensus.cc:348] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.318109 2186 raft_consensus.cc:374] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.318733 2186 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.319000 2390 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.319483 2186 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.319742 2390 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.322309 2390 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.326472 2186 ts_tablet_manager.cc:1428] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.063s user 0.011s sys 0.001s
I20260927 19:37:42.327920 2186 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:42.328982 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:42.330528 2187 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:42.330997 2187 ts_tablet_manager.cc:1397] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.083s user 0.039s sys 0.025s
I20260927 19:37:42.329107 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:42.345052 2037 leader_election.cc:341] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
W20260927 19:37:42.345969 2037 leader_election.cc:341] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.346349 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:42.379498 2550 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:42.383093 2389 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.383719 2389 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.386072 2389 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:42.418603 2390 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.419287 2390 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.422474 2390 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.435130 2187 raft_consensus.cc:348] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.435954 2187 raft_consensus.cc:374] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.436210 2187 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.436916 2187 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.448832 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:42.455873 2037 consensus_peers.cc:487] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 -> Peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Couldn't send request to peer 69f09866d0a1477585f05b2f82ff4c67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:42.456663 1996 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:42.458876 2111 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:42.459910 2389 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:42.460568 2389 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:42.461040 2389 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.459925 2186 tablet_bootstrap.cc:654] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.465436 2187 ts_tablet_manager.cc:1428] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.134s user 0.025s sys 0.056s
I20260927 19:37:42.464359 2550 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260927 19:37:42.472330 2187 tablet_bootstrap.cc:492] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:42.472455 2550 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.473181 2550 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.482350 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:42.484925 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:42.487094 2550 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.492316 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:42.500089 2517 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.501036 2517 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.503993 2517 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.504866 2389 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.512609 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:42.525344 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:42.528060 2389 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:42.528677 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:42.529305 1996 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.533504 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:37:42.538223 2037 leader_election.cc:341] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.541211 2185 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:42.541761 2185 ts_tablet_manager.cc:1397] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.298s user 0.012s sys 0.024s
I20260927 19:37:42.543435 1992 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:42.545075 2110 leader_election.cc:341] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.548455 1961 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:42.559930 2187 tablet_bootstrap.cc:654] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.565655 1996 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:42.566675 2186 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:42.567104 2186 ts_tablet_manager.cc:1397] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.239s user 0.018s sys 0.024s
I20260927 19:37:42.572368 2185 raft_consensus.cc:348] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:42.571168 2037 leader_election.cc:341] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.573163 2185 raft_consensus.cc:374] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.573431 2185 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.573269 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:42.574601 2111 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:42.575601 2185 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:42.577126 2037 leader_election.cc:341] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.574432 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:42.582727 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:42.582931 2185 ts_tablet_manager.cc:1428] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260927 19:37:42.583513 2142 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:42.584985 2387 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:42.578945 2550 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:42.584694 2185 tablet_bootstrap.cc:492] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:42.585841 2387 raft_consensus.cc:686] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:42.593240 2387 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.597956 2186 raft_consensus.cc:348] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.598814 2186 raft_consensus.cc:374] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260927 19:37:42.601136 2110 leader_election.cc:341] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.602830 2186 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.607003 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:42.607916 2186 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.614593 2575 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:42.621589 2575 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:42.622004 2575 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.624115 2186 ts_tablet_manager.cc:1428] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.057s user 0.006s sys 0.002s
I20260927 19:37:42.656684 2422 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:42.657279 2574 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.658037 2574 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.668179 2187 tablet_bootstrap.cc:492] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:42.668834 2187 ts_tablet_manager.cc:1397] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.197s user 0.012s sys 0.009s
I20260927 19:37:42.678216 2575 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.678704 2574 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.681628 2186 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:42.680763 2187 raft_consensus.cc:348] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.682140 2187 raft_consensus.cc:374] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.682423 2187 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.682988 2187 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.685014 2575 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.686208 2185 tablet_bootstrap.cc:654] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.691900 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:42.692349 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:42.693768 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:42.694464 2143 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:42.694927 2037 leader_election.cc:341] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.691972 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:42.696970 1992 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:42.700680 2187 ts_tablet_manager.cc:1428] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20260927 19:37:42.699747 1892 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:42.702116 2187 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:42.712852 2143 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:42.714066 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
W20260927 19:37:42.716212 2037 leader_election.cc:341] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.718274 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:37:42.719345 2576 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:42.719864 2576 raft_consensus.cc:686] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:42.720846 2576 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.724397 2187 tablet_bootstrap.cc:654] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.717257 2575 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:42.725906 2575 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:42.726423 2575 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.745843 2186 tablet_bootstrap.cc:654] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.755370 2185 tablet_bootstrap.cc:492] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:42.755885 2185 ts_tablet_manager.cc:1397] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.171s user 0.015s sys 0.005s
I20260927 19:37:42.814303 2524 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:42.814945 2524 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.809039 2185 raft_consensus.cc:348] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.850870 2185 raft_consensus.cc:374] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.852123 2185 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.853144 2185 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.871608 2524 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:42.879964 2575 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.882761 2186 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:42.883508 2186 ts_tablet_manager.cc:1397] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.228s user 0.015s sys 0.013s
I20260927 19:37:42.885830 2185 ts_tablet_manager.cc:1428] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.130s user 0.005s sys 0.008s
I20260927 19:37:42.882761 2187 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:42.886554 2187 ts_tablet_manager.cc:1397] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.185s user 0.012s sys 0.023s
I20260927 19:37:42.895459 2186 raft_consensus.cc:348] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.896034 2186 raft_consensus.cc:374] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.896241 2186 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
W20260927 19:37:42.897336 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47591.7us
I20260927 19:37:42.896749 2186 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.905048 2186 ts_tablet_manager.cc:1428] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.021s user 0.006s sys 0.002s
I20260927 19:37:42.908852 2186 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:42.909142 2575 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:42.910573 2185 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:42.912828 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:42.913522 1994 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:42.919059 2111 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:42.920104 2546 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:42.920482 2546 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:42.920794 2546 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.924072 2186 tablet_bootstrap.cc:654] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.935175 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:42.935884 2068 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:42.939239 2187 raft_consensus.cc:348] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.940203 2187 raft_consensus.cc:374] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:42.943583 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:42.946065 2187 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:42.946693 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:42.947784 2187 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:42.952674 2037 leader_election.cc:341] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:42.951627 1992 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.958886 2546 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.961995 2546 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:42.963680 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:42.964300 1994 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.964617 1992 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:42.965413 2187 ts_tablet_manager.cc:1428] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.079s user 0.007s sys 0.000s
I20260927 19:37:42.966976 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:42.967804 2469 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:42.968252 2469 raft_consensus.cc:686] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:42.969038 2185 tablet_bootstrap.cc:654] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:42.969055 2469 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.969732 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:42.971441 2063 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:42.975816 2187 tablet_bootstrap.cc:492] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:42.982519 1994 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:42.985491 2111 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:42.986732 2546 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:42.987178 2546 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:42.987948 2546 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:42.982657 2063 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:42.998198 2186 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:42.998780 2186 ts_tablet_manager.cc:1397] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.090s user 0.007s sys 0.006s
I20260927 19:37:43.001143 2186 raft_consensus.cc:348] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.001799 2186 raft_consensus.cc:374] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.002096 2186 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.002727 2186 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.002779 1893 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:43.006425 2186 ts_tablet_manager.cc:1428] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 19:37:43.007584 2186 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Bootstrap starting.
I20260927 19:37:43.011663 2546 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:43.016050 2185 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.016626 2185 ts_tablet_manager.cc:1397] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.116s user 0.013s sys 0.000s
I20260927 19:37:43.019217 2185 raft_consensus.cc:348] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.019939 2185 raft_consensus.cc:374] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.020222 2185 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.020931 2185 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.023864 2185 ts_tablet_manager.cc:1428] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260927 19:37:43.024879 2185 tablet_bootstrap.cc:492] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:43.030377 2185 tablet_bootstrap.cc:654] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.033661 2186 tablet_bootstrap.cc:654] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.049877 2187 tablet_bootstrap.cc:654] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.053777 1891 catalog_manager.cc:5495] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:43.064687 2185 tablet_bootstrap.cc:492] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.065660 2185 ts_tablet_manager.cc:1397] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.011s
I20260927 19:37:43.066783 2546 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.067268 2546 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.070777 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.073524 2576 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.074025 2576 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.076020 2576 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.077271 1893 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:43.079425 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:43.078595 2185 raft_consensus.cc:348] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.080104 2185 raft_consensus.cc:374] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.080909 2185 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
W20260927 19:37:43.081604 2037 leader_election.cc:341] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:43.081228 2186 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: No bootstrap required, opened a new log
I20260927 19:37:43.083003 2186 ts_tablet_manager.cc:1397] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.002s
I20260927 19:37:43.086171 2187 tablet_bootstrap.cc:492] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.086638 2187 ts_tablet_manager.cc:1397] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.112s user 0.008s sys 0.009s
I20260927 19:37:43.086452 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.081777 2185 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.090634 2185 ts_tablet_manager.cc:1428] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.025s user 0.004s sys 0.001s
I20260927 19:37:43.090938 2186 raft_consensus.cc:348] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.091946 2186 raft_consensus.cc:374] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.092092 2185 tablet_bootstrap.cc:492] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:43.092239 2186 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.090826 2187 raft_consensus.cc:348] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.093091 2187 raft_consensus.cc:374] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.093326 2187 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
W20260927 19:37:43.093945 2037 leader_election.cc:341] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:43.093902 2187 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.094424 2037 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:43.095100 2469 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:43.095645 2546 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.096086 2187 ts_tablet_manager.cc:1428] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260927 19:37:43.096643 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.097270 2186 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.098937 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:43.100148 2186 ts_tablet_manager.cc:1428] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260927 19:37:43.100973 2546 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:43.107765 2185 tablet_bootstrap.cc:654] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.111636 2187 tablet_bootstrap.cc:492] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:43.117280 2187 tablet_bootstrap.cc:654] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.128002 2185 tablet_bootstrap.cc:492] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.128625 2185 ts_tablet_manager.cc:1397] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.037s user 0.017s sys 0.009s
I20260927 19:37:43.129825 2063 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.137194 2387 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.138166 2185 raft_consensus.cc:348] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.138983 2185 raft_consensus.cc:374] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.139297 2185 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.140022 2185 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.142197 2185 ts_tablet_manager.cc:1428] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20260927 19:37:43.142763 2187 tablet_bootstrap.cc:492] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.143225 2187 ts_tablet_manager.cc:1397] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.007s
I20260927 19:37:43.143265 2185 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:43.145641 2187 raft_consensus.cc:348] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.146387 2187 raft_consensus.cc:374] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.146670 2187 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.147364 2187 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.149880 2187 ts_tablet_manager.cc:1428] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 19:37:43.151968 2185 tablet_bootstrap.cc:654] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.152863 2187 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:43.163971 2185 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.164499 2185 ts_tablet_manager.cc:1397] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.001s
I20260927 19:37:43.165105 2187 tablet_bootstrap.cc:654] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.165982 2142 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.168685 2185 raft_consensus.cc:348] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.169565 2185 raft_consensus.cc:374] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.169993 2185 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.170801 2185 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.173400 2185 ts_tablet_manager.cc:1428] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 19:37:43.175884 2185 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:43.176165 2187 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.176608 2187 ts_tablet_manager.cc:1397] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.000s
I20260927 19:37:43.180001 2187 raft_consensus.cc:348] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.180711 2187 raft_consensus.cc:374] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.180990 2187 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.182052 2185 tablet_bootstrap.cc:654] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.183858 2187 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.189182 2187 ts_tablet_manager.cc:1428] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260927 19:37:43.190372 2187 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:43.193317 2185 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.203616 2187 tablet_bootstrap.cc:654] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.203931 2185 ts_tablet_manager.cc:1397] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.003s
I20260927 19:37:43.207479 2387 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.208025 2387 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.210160 2387 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.212832 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.213737 1992 raft_consensus.cc:2379] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:37:43.214244 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.215392 2185 raft_consensus.cc:348] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.217128 2187 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.217427 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:43.217801 2187 ts_tablet_manager.cc:1397] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.005s
I20260927 19:37:43.218645 2389 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.217106 2185 raft_consensus.cc:374] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.221086 2185 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.221727 2185 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.221514 2187 raft_consensus.cc:348] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.223047 2187 raft_consensus.cc:374] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.225013 2185 ts_tablet_manager.cc:1428] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260927 19:37:43.225339 2187 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.227303 2187 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:37:43.228053 2037 consensus_peers.cc:487] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 -> Peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Couldn't send request to peer 3d1bc2cd9efe43399ddba396f5ddff85. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:43.229578 2187 ts_tablet_manager.cc:1428] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260927 19:37:43.230545 2187 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:43.232924 2185 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
I20260927 19:37:43.227466 2389 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:43.243534 2469 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.244033 2469 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.246639 2469 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.249485 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.250320 1992 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 0.
I20260927 19:37:43.250454 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:43.251385 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:43.252149 2469 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:43.252499 2469 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:43.252893 2185 tablet_bootstrap.cc:654] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.257946 2187 tablet_bootstrap.cc:654] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
W20260927 19:37:43.258770 2037 leader_election.cc:341] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:43.252804 2469 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.275287 2469 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.275988 2185 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.276464 2185 ts_tablet_manager.cc:1397] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.044s user 0.012s sys 0.000s
I20260927 19:37:43.277333 2469 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.277997 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:43.278430 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:43.278580 1992 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
W20260927 19:37:43.280292 2037 leader_election.cc:341] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:37:43.280305 2185 raft_consensus.cc:348] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.281068 2185 raft_consensus.cc:374] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.281400 2185 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.281976 2185 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.285831 1992 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:37:43.287833 2185 ts_tablet_manager.cc:1428] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260927 19:37:43.289868 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:43.292457 2469 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:43.294397 2469 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:37:43.295184 2469 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.295850 2185 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Bootstrap starting.
W20260927 19:37:43.301098 1942 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260927 19:37:43.303524 2576 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.318399 2185 tablet_bootstrap.cc:654] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.318986 2187 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.319661 2187 ts_tablet_manager.cc:1397] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.003s
I20260927 19:37:43.323046 2187 raft_consensus.cc:348] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.325654 2187 raft_consensus.cc:374] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.326280 2187 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.327093 2187 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.328962 2187 ts_tablet_manager.cc:1428] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260927 19:37:43.330013 2187 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:43.329314 1891 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:43.345309 2187 tablet_bootstrap.cc:654] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.345528 2185 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: No bootstrap required, opened a new log
I20260927 19:37:43.346055 2185 ts_tablet_manager.cc:1397] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.003s
I20260927 19:37:43.349833 2185 raft_consensus.cc:348] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.350589 2185 raft_consensus.cc:374] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.350934 2185 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.351581 2185 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.356442 2185 ts_tablet_manager.cc:1428] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260927 19:37:43.359023 2187 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.359465 2187 ts_tablet_manager.cc:1397] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.003s
I20260927 19:37:43.361742 2187 raft_consensus.cc:348] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.362447 2187 raft_consensus.cc:374] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.362686 2187 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.361698 2389 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.363216 2389 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.363251 2187 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.364954 2389 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.365512 2187 ts_tablet_manager.cc:1428] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 19:37:43.366430 2187 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Bootstrap starting.
I20260927 19:37:43.367281 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.367919 1992 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:43.369272 2111 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.369936 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.370836 2063 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:43.372521 2524 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:43.372638 2187 tablet_bootstrap.cc:654] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Neither blocks nor log segments found. Creating new log.
I20260927 19:37:43.372962 2524 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:43.373327 2524 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.380502 2546 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.381032 2546 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.383533 2546 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.389871 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.390582 2068 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:43.391299 2187 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: No bootstrap required, opened a new log
I20260927 19:37:43.391732 2187 ts_tablet_manager.cc:1397] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20260927 19:37:43.392184 2518 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.392661 2518 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.393987 2187 raft_consensus.cc:348] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.394537 2518 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.394613 2187 raft_consensus.cc:374] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 19:37:43.395015 2187 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Initialized, Role: FOLLOWER
I20260927 19:37:43.396093 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:43.396765 2187 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.397575 2142 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:43.398068 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.398677 1994 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:43.398801 2187 ts_tablet_manager.cc:1428] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:43.400151 1962 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.401106 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.401747 2063 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:43.401952 2110 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.402480 2518 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:43.402704 2389 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:43.402881 2518 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:43.403088 2389 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:43.403421 2518 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.403491 2389 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.409235 2518 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.409011 2524 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.411949 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:43.412271 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:43.412526 2063 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.412859 1994 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.417471 2389 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.419464 2063 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:43.419577 2389 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.421062 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:43.421202 2518 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.421726 2063 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.422540 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:43.423139 2142 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.426316 1961 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:37:43.426465 2063 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:43.427325 2592 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:43.427829 2592 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:43.428092 2110 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.428818 2389 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:43.429260 2389 raft_consensus.cc:686] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:43.429960 2389 consensus_queue.cc:237] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.430541 2592 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.436441 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:43.437099 1992 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.437335 2142 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:43.432703 1994 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:43.449213 1891 catalog_manager.cc:5495] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:43.451639 2387 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.452150 2387 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.454035 2387 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.457847 1992 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:43.459942 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.461136 1992 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:43.464521 2111 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.464572 2469 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.465549 2546 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:43.466096 2546 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:43.466596 2546 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.479351 2524 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.481230 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.481952 2063 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 0.
I20260927 19:37:43.483803 2111 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.486263 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:43.486838 2063 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.485531 2524 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:43.491264 2063 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:43.495133 2546 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.497035 2524 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:43.496778 1891 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:43.497855 2524 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.499264 2546 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.500195 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:37:43.500816 1992 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.505756 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:43.506382 2063 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.511940 2592 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.512429 2592 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.514976 2063 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:43.516160 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:43.517024 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.517423 2592 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.520020 1992 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:37:43.522030 2546 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:43.527554 2110 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.528400 2546 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:43.528962 2546 raft_consensus.cc:686] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:37:43.530158 2546 consensus_queue.cc:237] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.531545 1892 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:43.533954 2469 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.536039 1961 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:37:43.537233 2527 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:43.558409 2387 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.562047 2592 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.562652 2592 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.564991 2592 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.565915 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:37:43.566586 2141 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:43.568092 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.568717 1962 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.568948 2066 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 0.
I20260927 19:37:43.569805 2592 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 19:37:43.570185 2592 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:37:43.570580 2592 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.580832 2592 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.583904 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:37:43.584501 2141 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.587688 1891 catalog_manager.cc:5495] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:37:43.591831 2141 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:43.594795 2592 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:43.596282 1962 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:37:43.596698 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:37:43.597270 2068 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.598443 2527 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 19:37:43.598925 2527 raft_consensus.cc:686] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:37:43.599727 2527 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.604449 2068 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:37:43.621973 1891 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:43.695704 1992 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.713012 2524 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:37:43.713560 2524 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:37:43.715780 2524 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:43.716235 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:37:43.716984 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:37:43.718444 1992 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 1: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:37:43.719488 2111 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:37:43.720296 2524 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 19:37:43.725968 2524 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 19:37:43.820261 2469 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.831477 2588 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.840765 2524 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.857493 2387 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.868538 2527 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.896978 2517 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.908298 2389 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 19:37:43.939216 2387 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.966492 2387 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.979596 2389 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:43.984773 2517 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:44.014496 2517 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 19:37:44.522522 818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 19:37:44.536785 2642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:44.544533 2643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 19:37:44.559861 2645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 19:37:44.560065 818 server_base.cc:1034] running on GCE node
I20260927 19:37:44.562585 818 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260927 19:37:44.562886 818 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260927 19:37:44.563083 818 hybrid_clock.cc:648] HybridClock initialized: now 1790537864563066 us; error 0 us; skew 500 ppm
I20260927 19:37:44.563781 818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 19:37:44.573838 818 webserver.cc:460] Webserver started at http://127.0.204.132:43797/ using document root <none> and password file <none>
I20260927 19:37:44.574447 818 fs_manager.cc:359] Metadata directory not provided
I20260927 19:37:44.574685 818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 19:37:44.575109 818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 19:37:44.576838 818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-3-root/instance:
uuid: "4e0335d1a85f421380029b95a91bf0c4"
format_stamp: "Formatted at 2026-09-27 19:37:44 on dist-test-slave-zwk0"
I20260927 19:37:44.582705 818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 19:37:44.586880 2650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:44.587662 818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 19:37:44.587994 818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-3-root
uuid: "4e0335d1a85f421380029b95a91bf0c4"
format_stamp: "Formatted at 2026-09-27 19:37:44 on dist-test-slave-zwk0"
I20260927 19:37:44.588308 818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 19:37:44.609300 818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 19:37:44.610776 818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 19:37:44.612632 818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 19:37:44.615298 818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 19:37:44.615669 818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:44.616003 818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 19:37:44.616195 818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 19:37:44.697184 818 rpc_server.cc:307] RPC server started. Bound to: 127.0.204.132:45629
I20260927 19:37:44.697286 2712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.204.132:45629 every 8 connection(s)
I20260927 19:37:44.709810 818 auto_leader_rebalancer-test.cc:413] add a tserver: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:44.724293 2713 heartbeater.cc:347] Connected to a master server at 127.0.204.190:41277
I20260927 19:37:44.724766 2713 heartbeater.cc:464] Registering TS with master...
I20260927 19:37:44.725615 2713 heartbeater.cc:511] Master 127.0.204.190:41277 requested a full tablet report, sending...
I20260927 19:37:44.727603 1892 ts_manager.cc:194] Registered new tserver with Master: 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:37:44.729204 1892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34824
I20260927 19:37:45.440527 2066 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:45.448136 2142 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:45.450138 2588 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:45.461788 1992 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:45.466804 2576 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:45.478760 2141 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.477752 2588 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
W20260927 19:37:45.502779 2034 consensus_peers.cc:487] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85f45e6282f545f19d655ea989e9e123. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:45.502398 1992 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.513629 1892 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.557543 1992 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:45.567564 2139 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:45.569026 2518 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:45.569300 2063 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:45.570649 2527 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 19:37:45.582706 1959 consensus_peers.cc:487] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aabcfbc05a724a19857afe1f8211259f. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:45.584560 2518 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.592202 2063 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.596498 2139 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.601290 1992 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:45.614427 1893 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260927 19:37:45.626432 1959 consensus_peers.cc:487] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2f6360a6b784953bc89cd72712a5e7d. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:45.627136 2139 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:45.629078 2735 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:45.629245 2063 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:45.631484 2518 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:45.648023 2735 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.651736 2070 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.654496 2139 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.665014 1892 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:45.685911 2139 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:45.693661 1992 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:45.694129 2070 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:37:45.694758 2389 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:45.701184 2725 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:45.703526 2546 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.708603 1992 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
W20260927 19:37:45.715617 2108 consensus_peers.cc:487] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63522eab31e04c8191338bb22acb0297. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:45.717257 2070 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.725560 1893 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 3, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:45.732154 2713 heartbeater.cc:503] Master 127.0.204.190:41277 was elected leader, sending a full tablet report...
I20260927 19:37:45.874382 2744 ts_tablet_manager.cc:927] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:45.877177 2744 tablet_copy_client.cc:316] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:45.896927 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet 85f45e6282f545f19d655ea989e9e123 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:45.897558 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet 85f45e6282f545f19d655ea989e9e123 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-85f45e6282f545f19d655ea989e9e123
I20260927 19:37:45.904582 2081 tablet_copy_source_session.cc:205] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:45.909518 2744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f45e6282f545f19d655ea989e9e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:45.923682 2744 tablet_copy_client.cc:791] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:45.924310 2744 tablet_copy_client.cc:655] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:45.928678 2744 tablet_copy_client.cc:531] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:45.936964 2744 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:45.957170 2744 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:45.958017 2744 tablet_bootstrap.cc:492] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:45.958683 2744 ts_tablet_manager.cc:1397] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.007s
I20260927 19:37:45.961696 2744 raft_consensus.cc:348] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:45.962493 2744 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:45.963349 2744 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:45.970942 2744 ts_tablet_manager.cc:1428] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260927 19:37:45.972744 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-85f45e6282f545f19d655ea989e9e123 received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:45.973187 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-85f45e6282f545f19d655ea989e9e123 on tablet 85f45e6282f545f19d655ea989e9e123 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:46.017791 2744 ts_tablet_manager.cc:927] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:46.019559 2744 tablet_copy_client.cc:316] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:46.037520 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet aabcfbc05a724a19857afe1f8211259f from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:46.038178 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet aabcfbc05a724a19857afe1f8211259f from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-aabcfbc05a724a19857afe1f8211259f
I20260927 19:37:46.048358 2006 tablet_copy_source_session.cc:205] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:46.051834 2744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabcfbc05a724a19857afe1f8211259f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:46.067323 2744 tablet_copy_client.cc:791] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:46.067927 2744 tablet_copy_client.cc:655] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:46.072762 2744 tablet_copy_client.cc:531] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:46.080592 2750 ts_tablet_manager.cc:927] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:46.085780 2750 tablet_copy_client.cc:316] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:37:46.088845 2744 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:46.101760 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet 63522eab31e04c8191338bb22acb0297 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:37:46.102514 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet 63522eab31e04c8191338bb22acb0297 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-63522eab31e04c8191338bb22acb0297
I20260927 19:37:46.109818 2155 tablet_copy_source_session.cc:205] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:46.110415 2744 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:46.111380 2744 tablet_bootstrap.cc:492] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:46.112053 2744 ts_tablet_manager.cc:1397] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20260927 19:37:46.112987 2750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63522eab31e04c8191338bb22acb0297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:46.114499 2744 raft_consensus.cc:348] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:46.115363 2744 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:46.116106 2744 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:46.118371 2744 ts_tablet_manager.cc:1428] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 19:37:46.123584 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-aabcfbc05a724a19857afe1f8211259f received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:46.123983 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-aabcfbc05a724a19857afe1f8211259f on tablet aabcfbc05a724a19857afe1f8211259f with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:46.137667 2750 tablet_copy_client.cc:791] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:46.138373 2750 tablet_copy_client.cc:655] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:46.142288 2750 tablet_copy_client.cc:531] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:46.152107 2750 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:46.152211 2744 ts_tablet_manager.cc:927] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:46.154101 2744 tablet_copy_client.cc:316] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:46.156083 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet b2f6360a6b784953bc89cd72712a5e7d from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:46.156673 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet b2f6360a6b784953bc89cd72712a5e7d from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-b2f6360a6b784953bc89cd72712a5e7d
I20260927 19:37:46.175783 2006 tablet_copy_source_session.cc:205] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:46.179088 2744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2f6360a6b784953bc89cd72712a5e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:46.191850 2750 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:46.192943 2750 tablet_bootstrap.cc:492] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:46.193742 2750 ts_tablet_manager.cc:1397] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.042s user 0.028s sys 0.012s
I20260927 19:37:46.195416 2744 tablet_copy_client.cc:791] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:46.196321 2744 tablet_copy_client.cc:655] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:46.197000 2750 raft_consensus.cc:348] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:46.198078 2750 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:46.198704 2750 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:46.201592 2750 ts_tablet_manager.cc:1428] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 19:37:46.203119 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-63522eab31e04c8191338bb22acb0297 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:37:46.203531 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-63522eab31e04c8191338bb22acb0297 on tablet 63522eab31e04c8191338bb22acb0297 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:46.208693 2744 tablet_copy_client.cc:531] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:46.219635 2744 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:46.235373 2744 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:46.236596 2744 tablet_bootstrap.cc:492] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:46.237174 2744 ts_tablet_manager.cc:1397] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260927 19:37:46.239759 2744 raft_consensus.cc:348] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:46.240538 2744 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:46.241079 2744 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:46.244015 2744 ts_tablet_manager.cc:1428] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260927 19:37:46.245635 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-b2f6360a6b784953bc89cd72712a5e7d received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:46.246052 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-b2f6360a6b784953bc89cd72712a5e7d on tablet b2f6360a6b784953bc89cd72712a5e7d with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:46.460084 2688 raft_consensus.cc:1203] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20260927 19:37:46.484236 2688 raft_consensus.cc:1203] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:46.588176 2688 raft_consensus.cc:1203] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:46.710642 2688 raft_consensus.cc:1203] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:46.967291 2756 raft_consensus.cc:1050] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:46.969823 2756 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:46.976157 2068 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260927 19:37:46.976658 1994 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260927 19:37:46.977202 2688 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260927 19:37:46.978644 2546 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:46.979547 2389 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:46.980268 2756 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:46.996961 2546 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:46.999245 2066 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.000691 1994 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.017055 2688 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.024585 1893 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 3 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:47.043638 2588 raft_consensus.cc:1050] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:47.046335 2588 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.054548 2141 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.055949 2139 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.057663 2688 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.059389 2764 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:47.063141 2764 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:47.065121 1996 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.067179 2588 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:47.084453 2687 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:37:47.088207 2546 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:37:47.095896 2068 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:37:47.097170 2762 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.098518 2687 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.110711 2389 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:37:47.115096 2068 raft_consensus.cc:2932] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.116835 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 63522eab31e04c8191338bb22acb0297 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:47.118014 1892 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 4 to 5, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.138168 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 63522eab31e04c8191338bb22acb0297 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60280
I20260927 19:37:47.154345 2720 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.175096 1994 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.182476 2140 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.184229 2772 tablet_replica.cc:330] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:37:47.188549 2772 raft_consensus.cc:2229] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:47.189428 2772 raft_consensus.cc:2258] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:47.191696 2688 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.194852 2772 ts_tablet_manager.cc:1905] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:47.196599 1892 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.216180 2518 raft_consensus.cc:1050] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:47.218406 2518 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.235248 2688 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.237192 2527 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:47.244098 2140 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.245735 2527 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:47.254354 2772 ts_tablet_manager.cc:1918] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260927 19:37:47.255105 2772 log.cc:1196] T 63522eab31e04c8191338bb22acb0297 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/63522eab31e04c8191338bb22acb0297
I20260927 19:37:47.256994 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 63522eab31e04c8191338bb22acb0297 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:47.260383 2070 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.262207 2527 raft_consensus.cc:1050] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:47.267881 2688 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.270367 2527 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.272533 2518 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:47.265555 2735 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.280874 2068 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.289000 2140 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:47.287963 1890 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.290102 2688 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:47.293825 2588 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:47.296160 2588 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:47.304834 2139 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.310973 2773 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.318397 2688 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.326977 2070 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.335045 1996 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.337415 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 85f45e6282f545f19d655ea989e9e123 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:47.342454 2139 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:47.344504 2687 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:47.346175 2142 raft_consensus.cc:2932] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.350106 1892 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.352075 2518 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:47.354285 2735 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:47.364347 2517 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.368074 2141 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.367547 2687 raft_consensus.cc:2932] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.381901 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet aabcfbc05a724a19857afe1f8211259f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:47.381918 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f45e6282f545f19d655ea989e9e123 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:37:47.383462 2772 tablet_replica.cc:330] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:37:47.384810 2772 raft_consensus.cc:2229] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:47.385742 2772 raft_consensus.cc:2258] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:47.395308 2772 ts_tablet_manager.cc:1905] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:47.399470 1890 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:47.406543 2688 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.409020 2735 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:47.415143 2139 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.416767 2517 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:47.422613 2772 ts_tablet_manager.cc:1918] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:47.423067 2772 log.cc:1196] T 85f45e6282f545f19d655ea989e9e123 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/85f45e6282f545f19d655ea989e9e123
I20260927 19:37:47.428169 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 85f45e6282f545f19d655ea989e9e123 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:47.435273 2735 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.436779 2142 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.440420 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet aabcfbc05a724a19857afe1f8211259f with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:37:47.442754 2070 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:47.449863 2527 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:47.476610 1893 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:37:47.497444 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44716us
I20260927 19:37:47.481305 2781 tablet_replica.cc:330] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:37:47.470366 2688 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:47.509721 2063 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:37:47.544031 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 91394.5us
I20260927 19:37:47.568815 2781 raft_consensus.cc:2229] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:47.569486 2781 raft_consensus.cc:2258] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:47.572463 2781 ts_tablet_manager.cc:1905] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:47.585074 2781 ts_tablet_manager.cc:1918] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:47.585479 2781 log.cc:1196] T aabcfbc05a724a19857afe1f8211259f P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/aabcfbc05a724a19857afe1f8211259f
I20260927 19:37:47.586985 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet aabcfbc05a724a19857afe1f8211259f (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:47.980417 1996 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:47.985719 2066 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:47.986377 2688 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:47.987175 2783 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:47.988087 2735 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:47.998767 2735 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:48.006963 2063 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:48.010269 2688 raft_consensus.cc:2932] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:48.021952 1891 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:48.023671 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b2f6360a6b784953bc89cd72712a5e7d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:48.040606 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet b2f6360a6b784953bc89cd72712a5e7d with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:37:48.042493 2787 tablet_replica.cc:330] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:37:48.043567 2787 raft_consensus.cc:2229] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:48.048105 2787 raft_consensus.cc:2258] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:48.051620 2787 ts_tablet_manager.cc:1905] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:48.064851 2787 ts_tablet_manager.cc:1918] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:48.065162 2787 log.cc:1196] T b2f6360a6b784953bc89cd72712a5e7d P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/b2f6360a6b784953bc89cd72712a5e7d
I20260927 19:37:48.066594 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet b2f6360a6b784953bc89cd72712a5e7d (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:49.123565 1995 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.130414 2141 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.131961 2735 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260927 19:37:49.135135 1959 consensus_peers.cc:487] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 303e2824ebc54b4c9bce740f15e0105d. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:49.137477 2063 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.141812 2735 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.143432 2790 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.147222 2141 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.156152 2066 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.167380 2063 consensus_queue.cc:237] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.171404 1893 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260927 19:37:49.186730 2034 consensus_peers.cc:487] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efcb4aec6b174f11b08d4483d9f63866. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:49.187395 1995 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.185393 2141 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.188676 2788 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.190568 2588 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.201411 2588 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.202458 2141 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.202979 1995 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.221184 2141 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.222419 1893 catalog_manager.cc:5495] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
W20260927 19:37:49.229765 2108 consensus_peers.cc:487] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: acb3a2a0b9b047249cd5a7535a946fe9. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:49.232155 2063 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.232215 1995 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.233520 2546 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.236274 2758 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.243180 2758 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.244566 2063 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.253250 1995 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.272203 1995 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.274101 1891 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.287564 2141 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:49.288290 1959 consensus_peers.cc:487] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0505ba7a2573409cbd518d77ccbcd201. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:49.288983 2790 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.289295 2068 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:49.291299 2790 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:49.304278 2735 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.306380 2142 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.310719 2068 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.329830 1893 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:49.514889 2809 ts_tablet_manager.cc:927] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:49.522513 2809 tablet_copy_client.cc:316] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:49.524227 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet 303e2824ebc54b4c9bce740f15e0105d from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:49.524806 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet 303e2824ebc54b4c9bce740f15e0105d from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-303e2824ebc54b4c9bce740f15e0105d
I20260927 19:37:49.532339 2006 tablet_copy_source_session.cc:205] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:49.535595 2809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 303e2824ebc54b4c9bce740f15e0105d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:49.551926 2809 tablet_copy_client.cc:791] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:49.552542 2809 tablet_copy_client.cc:655] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:49.556344 2809 tablet_copy_client.cc:531] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:49.564826 2809 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:49.587291 2809 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:49.588039 2809 tablet_bootstrap.cc:492] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:49.588572 2809 ts_tablet_manager.cc:1397] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.001s
I20260927 19:37:49.590916 2809 raft_consensus.cc:348] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.591553 2809 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:49.592044 2809 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.595001 2809 ts_tablet_manager.cc:1428] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260927 19:37:49.596835 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-303e2824ebc54b4c9bce740f15e0105d received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:49.597203 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-303e2824ebc54b4c9bce740f15e0105d on tablet 303e2824ebc54b4c9bce740f15e0105d with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:49.698412 2809 ts_tablet_manager.cc:927] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:49.708026 2809 tablet_copy_client.cc:316] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:49.709620 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet efcb4aec6b174f11b08d4483d9f63866 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:49.710098 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet efcb4aec6b174f11b08d4483d9f63866 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-efcb4aec6b174f11b08d4483d9f63866
I20260927 19:37:49.716224 2081 tablet_copy_source_session.cc:205] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:49.719291 2809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efcb4aec6b174f11b08d4483d9f63866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:49.733918 2809 tablet_copy_client.cc:791] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:49.734544 2809 tablet_copy_client.cc:655] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:49.738538 2809 tablet_copy_client.cc:531] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:49.738803 2814 ts_tablet_manager.cc:927] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:49.740901 2814 tablet_copy_client.cc:316] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:49.742306 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet 0505ba7a2573409cbd518d77ccbcd201 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:49.742801 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet 0505ba7a2573409cbd518d77ccbcd201 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-0505ba7a2573409cbd518d77ccbcd201
I20260927 19:37:49.748494 2809 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:49.750710 2006 tablet_copy_source_session.cc:205] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:49.758006 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0505ba7a2573409cbd518d77ccbcd201. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:49.769614 2809 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:49.770668 2809 tablet_bootstrap.cc:492] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:49.771355 2809 ts_tablet_manager.cc:1397] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.001s
I20260927 19:37:49.774186 2809 raft_consensus.cc:348] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.775067 2809 raft_consensus.cc:729] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:49.775736 2809 consensus_queue.cc:260] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.778151 2809 ts_tablet_manager.cc:1428] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 19:37:49.779600 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-efcb4aec6b174f11b08d4483d9f63866 received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:49.779980 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-efcb4aec6b174f11b08d4483d9f63866 on tablet efcb4aec6b174f11b08d4483d9f63866 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:49.780335 2814 tablet_copy_client.cc:791] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:49.780944 2814 tablet_copy_client.cc:655] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:49.788347 2814 tablet_copy_client.cc:531] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:49.796072 2814 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:49.818845 2814 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:49.819625 2814 tablet_bootstrap.cc:492] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:49.820135 2814 ts_tablet_manager.cc:1397] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20260927 19:37:49.822409 2814 raft_consensus.cc:348] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.823088 2814 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:49.823675 2814 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.826073 2814 ts_tablet_manager.cc:1428] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 19:37:49.830946 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-0505ba7a2573409cbd518d77ccbcd201 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:49.831338 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-0505ba7a2573409cbd518d77ccbcd201 on tablet 0505ba7a2573409cbd518d77ccbcd201 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:49.847337 2814 ts_tablet_manager.cc:927] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:49.850395 2814 tablet_copy_client.cc:316] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:37:49.851869 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet acb3a2a0b9b047249cd5a7535a946fe9 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:37:49.852357 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet acb3a2a0b9b047249cd5a7535a946fe9 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-acb3a2a0b9b047249cd5a7535a946fe9
I20260927 19:37:49.859581 2155 tablet_copy_source_session.cc:205] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:49.862541 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb3a2a0b9b047249cd5a7535a946fe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:49.876288 2814 tablet_copy_client.cc:791] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:49.876818 2814 tablet_copy_client.cc:655] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:49.882647 2814 tablet_copy_client.cc:531] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:49.893978 2814 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:49.902872 2688 raft_consensus.cc:1203] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:49.921511 2814 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:49.922458 2814 tablet_bootstrap.cc:492] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:49.923213 2814 ts_tablet_manager.cc:1397] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.011s
I20260927 19:37:49.926263 2814 raft_consensus.cc:348] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.927153 2814 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:49.927784 2814 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:49.930199 2814 ts_tablet_manager.cc:1428] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20260927 19:37:49.931968 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-acb3a2a0b9b047249cd5a7535a946fe9 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:37:49.932367 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-acb3a2a0b9b047249cd5a7535a946fe9 on tablet acb3a2a0b9b047249cd5a7535a946fe9 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:50.161293 2688 raft_consensus.cc:1203] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:50.222684 2688 raft_consensus.cc:1203] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:50.356526 2688 raft_consensus.cc:1203] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:50.375087 2527 raft_consensus.cc:1050] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:50.377140 2527 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.396533 2688 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.396857 2139 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.398166 2527 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.399730 2735 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.399955 2063 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.427424 2527 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.430183 2142 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.432216 2783 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260927 19:37:50.446660 2688 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.456033 1890 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:50.465579 2068 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.498497 1996 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.505604 2688 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:50.507606 2142 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:50.507434 2527 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:50.508932 2822 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:50.513882 2735 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.517747 2688 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.518899 2141 raft_consensus.cc:2932] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.524684 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 303e2824ebc54b4c9bce740f15e0105d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:50.533458 1892 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.548959 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet 303e2824ebc54b4c9bce740f15e0105d with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:37:50.555824 2831 tablet_replica.cc:330] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:37:50.556825 2831 raft_consensus.cc:2229] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:50.557523 2831 raft_consensus.cc:2258] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:50.561290 2831 ts_tablet_manager.cc:1905] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:50.577051 2831 ts_tablet_manager.cc:1918] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:50.577572 2831 log.cc:1196] T 303e2824ebc54b4c9bce740f15e0105d P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/303e2824ebc54b4c9bce740f15e0105d
I20260927 19:37:50.579025 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet 303e2824ebc54b4c9bce740f15e0105d (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:50.587914 2800 raft_consensus.cc:1050] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:50.589972 2800 consensus_queue.cc:237] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.597249 2688 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.598121 2141 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.598664 2788 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.600031 2833 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.617233 1995 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.620144 2800 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.621702 2788 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.627007 2688 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.635104 1995 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.635458 2141 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.643735 1891 catalog_manager.cc:5495] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:50.743221 2783 raft_consensus.cc:1050] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:50.745723 2783 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.752993 2688 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.754163 2068 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.755422 2822 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:50.754740 2141 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.756783 2735 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.762167 2822 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.783361 2804 raft_consensus.cc:1050] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:50.787672 2804 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.794000 2822 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.794903 2688 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.801390 2063 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.803334 2066 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.805315 2820 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.806198 2687 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.808588 2820 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:50.809691 1992 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:50.811138 2141 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.820080 2820 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:50.821717 1892 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:50.842700 1992 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.855499 2687 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:50.857214 2822 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:50.863318 2141 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:50.865823 2822 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:50.874406 2527 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.877714 2688 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.882403 2820 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.884500 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0505ba7a2573409cbd518d77ccbcd201 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:50.888697 2066 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.891023 1992 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.902717 1890 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.904098 2142 raft_consensus.cc:2932] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.919806 2688 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.925695 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet 0505ba7a2573409cbd518d77ccbcd201 with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:37:50.926757 2831 tablet_replica.cc:330] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:37:50.927757 2831 raft_consensus.cc:2229] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:50.928427 2831 raft_consensus.cc:2258] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:50.930342 1890 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.936563 2831 ts_tablet_manager.cc:1905] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:50.964419 2831 ts_tablet_manager.cc:1918] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:50.964906 2831 log.cc:1196] T 0505ba7a2573409cbd518d77ccbcd201 P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/0505ba7a2573409cbd518d77ccbcd201
I20260927 19:37:50.966820 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet 0505ba7a2573409cbd518d77ccbcd201 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:50.974238 2141 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:50.979005 2688 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:50.980315 2063 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:50.980324 2845 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:50.981707 2820 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:50.989495 2804 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.991750 2066 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:50.998278 2687 raft_consensus.cc:2932] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:51.006119 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet acb3a2a0b9b047249cd5a7535a946fe9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:51.010020 1890 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:51.021733 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet acb3a2a0b9b047249cd5a7535a946fe9 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:37:51.024432 2846 tablet_replica.cc:330] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:37:51.025496 2846 raft_consensus.cc:2229] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:51.026304 2846 raft_consensus.cc:2258] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:51.029227 2846 ts_tablet_manager.cc:1905] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:51.045918 2846 ts_tablet_manager.cc:1918] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:51.046311 2846 log.cc:1196] T acb3a2a0b9b047249cd5a7535a946fe9 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/acb3a2a0b9b047249cd5a7535a946fe9
I20260927 19:37:51.048089 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet acb3a2a0b9b047249cd5a7535a946fe9 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:51.110414 2066 consensus_queue.cc:237] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:51.115644 2687 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:51.116221 1996 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:51.117192 2800 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:51.118163 2788 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:51.125293 2833 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:51.126930 2687 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:51.137023 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet efcb4aec6b174f11b08d4483d9f63866 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:51.140404 1891 catalog_manager.cc:5495] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:51.142266 1992 raft_consensus.cc:2932] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:51.155282 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet efcb4aec6b174f11b08d4483d9f63866 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:37:51.156862 2847 tablet_replica.cc:330] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:37:51.157806 2847 raft_consensus.cc:2229] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:51.158479 2847 raft_consensus.cc:2258] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:51.162480 2847 ts_tablet_manager.cc:1905] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:51.182189 2847 ts_tablet_manager.cc:1918] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:51.182595 2847 log.cc:1196] T efcb4aec6b174f11b08d4483d9f63866 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/efcb4aec6b174f11b08d4483d9f63866
I20260927 19:37:51.184223 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet efcb4aec6b174f11b08d4483d9f63866 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:52.210423 2068 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:52.217491 2139 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:52.219545 2034 consensus_peers.cc:487] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d811e58acb4b4a3aa93ab155fd5ab1c2. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:52.220908 2788 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.232012 2853 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.233678 2141 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.296121 2063 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:52.332577 1996 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.323433 1892 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:52.350049 2833 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.367940 1996 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
W20260927 19:37:52.376461 2034 consensus_peers.cc:487] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e017d73273c4a83bcfdb4c95caa32df. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:52.394548 2141 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.409387 1995 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.410583 2788 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.428192 2853 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.456704 2833 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.466226 1996 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.462738 2141 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.490041 1995 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:37:52.506987 1959 consensus_peers.cc:487] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac63cb1bcdd141bb9de9fcb5d0f102af. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:52.500984 1891 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.516768 2141 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.518684 2856 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.531925 2070 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.534380 2855 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.546285 2856 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.547837 2141 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.558055 2070 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.563920 1890 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.564144 2071 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:37:52.584982 2034 consensus_peers.cc:487] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fe684e9563d491b9d4b3b99204450c8. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:52.589553 1995 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.589699 2139 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:52.591670 2788 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:52.592674 2854 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:52.606808 2788 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.608673 1995 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.611449 2139 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:52.632879 1892 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:52.835009 2874 ts_tablet_manager.cc:927] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:52.839793 2874 tablet_copy_client.cc:316] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:52.842098 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet d811e58acb4b4a3aa93ab155fd5ab1c2 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:52.842713 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet d811e58acb4b4a3aa93ab155fd5ab1c2 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-d811e58acb4b4a3aa93ab155fd5ab1c2
I20260927 19:37:52.846639 2876 ts_tablet_manager.cc:927] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:52.854988 2081 tablet_copy_source_session.cc:205] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:52.855350 2876 tablet_copy_client.cc:316] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:52.856994 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet 9e017d73273c4a83bcfdb4c95caa32df from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:52.857476 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet 9e017d73273c4a83bcfdb4c95caa32df from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-9e017d73273c4a83bcfdb4c95caa32df
I20260927 19:37:52.858273 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d811e58acb4b4a3aa93ab155fd5ab1c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:52.868562 2081 tablet_copy_source_session.cc:205] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:52.873311 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e017d73273c4a83bcfdb4c95caa32df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:52.879071 2874 tablet_copy_client.cc:791] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:52.879894 2874 tablet_copy_client.cc:655] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:52.886083 2874 tablet_copy_client.cc:531] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:52.890036 2876 tablet_copy_client.cc:791] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:52.890762 2876 tablet_copy_client.cc:655] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:52.894779 2876 tablet_copy_client.cc:531] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:52.896313 2874 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:52.905608 2876 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:52.931010 2876 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:52.931421 2874 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:52.932260 2876 tablet_bootstrap.cc:492] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:52.932967 2876 ts_tablet_manager.cc:1397] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260927 19:37:52.932265 2874 tablet_bootstrap.cc:492] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:52.934585 2874 ts_tablet_manager.cc:1397] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.039s user 0.029s sys 0.004s
I20260927 19:37:52.936568 2876 raft_consensus.cc:348] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:52.937678 2876 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:52.937484 2874 raft_consensus.cc:348] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:52.938501 2874 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:52.938442 2876 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:52.939325 2874 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:52.949781 2876 ts_tablet_manager.cc:1428] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.016s user 0.000s sys 0.015s
I20260927 19:37:52.952164 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-9e017d73273c4a83bcfdb4c95caa32df received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:52.952561 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-9e017d73273c4a83bcfdb4c95caa32df on tablet 9e017d73273c4a83bcfdb4c95caa32df with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:52.961519 2874 ts_tablet_manager.cc:1428] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.027s user 0.015s sys 0.005s
I20260927 19:37:52.963627 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-d811e58acb4b4a3aa93ab155fd5ab1c2 received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:52.964051 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-d811e58acb4b4a3aa93ab155fd5ab1c2 on tablet d811e58acb4b4a3aa93ab155fd5ab1c2 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:53.093647 2874 ts_tablet_manager.cc:927] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:53.095670 2874 tablet_copy_client.cc:316] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:53.097553 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet ac63cb1bcdd141bb9de9fcb5d0f102af from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:53.098037 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet ac63cb1bcdd141bb9de9fcb5d0f102af from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-ac63cb1bcdd141bb9de9fcb5d0f102af
I20260927 19:37:53.105599 2006 tablet_copy_source_session.cc:205] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:53.109972 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac63cb1bcdd141bb9de9fcb5d0f102af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:53.126958 2874 tablet_copy_client.cc:791] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:53.127580 2874 tablet_copy_client.cc:655] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:53.131250 2874 tablet_copy_client.cc:531] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:53.139588 2874 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:53.158174 2874 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:53.159067 2874 tablet_bootstrap.cc:492] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:53.159654 2874 ts_tablet_manager.cc:1397] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260927 19:37:53.162104 2874 raft_consensus.cc:348] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:53.162863 2874 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:53.163425 2874 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:53.165585 2874 ts_tablet_manager.cc:1428] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260927 19:37:53.167388 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-ac63cb1bcdd141bb9de9fcb5d0f102af received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:53.167814 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-ac63cb1bcdd141bb9de9fcb5d0f102af on tablet ac63cb1bcdd141bb9de9fcb5d0f102af with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:53.170065 2876 ts_tablet_manager.cc:927] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:53.171830 2876 tablet_copy_client.cc:316] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:53.173426 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet 4fe684e9563d491b9d4b3b99204450c8 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:53.173979 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet 4fe684e9563d491b9d4b3b99204450c8 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-4fe684e9563d491b9d4b3b99204450c8
I20260927 19:37:53.189337 2081 tablet_copy_source_session.cc:205] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:53.192737 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fe684e9563d491b9d4b3b99204450c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:53.207749 2876 tablet_copy_client.cc:791] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:53.208331 2876 tablet_copy_client.cc:655] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:53.212656 2876 tablet_copy_client.cc:531] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:53.222338 2876 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:53.243885 2876 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:53.244987 2876 tablet_bootstrap.cc:492] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:53.245731 2876 ts_tablet_manager.cc:1397] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.005s
I20260927 19:37:53.278016 2876 raft_consensus.cc:348] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:53.278677 2688 raft_consensus.cc:1203] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:53.280246 2876 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:53.288997 2876 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:53.295367 2876 ts_tablet_manager.cc:1428] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.049s user 0.009s sys 0.000s
I20260927 19:37:53.348433 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-4fe684e9563d491b9d4b3b99204450c8 received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:53.348968 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-4fe684e9563d491b9d4b3b99204450c8 on tablet 4fe684e9563d491b9d4b3b99204450c8 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:53.369853 2688 raft_consensus.cc:1203] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:53.627720 2688 raft_consensus.cc:1203] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:53.637562 2688 raft_consensus.cc:1203] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:53.879202 2854 raft_consensus.cc:1050] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:53.881572 2854 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:53.891662 2139 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:53.893913 2688 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:53.895448 2853 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:53.897146 1995 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:53.899544 2788 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:53.903012 2788 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:53.932994 2853 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:53.944927 2688 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:53.951022 2139 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:53.959934 2854 raft_consensus.cc:1050] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:53.963677 2854 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:53.969614 1996 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:53.987501 1892 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.017040 2688 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.019536 2139 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.020364 1995 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.022260 2892 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:54.023582 2788 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:54.027944 2853 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:54.056488 2853 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.058001 2688 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.062089 1995 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.080161 2071 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:54.088264 2139 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.094714 2688 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.097076 2892 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:54.108179 1995 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.114885 2788 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:54.118485 2854 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.126796 2688 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.092330 1890 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.137653 1995 raft_consensus.cc:2932] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.146445 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9e017d73273c4a83bcfdb4c95caa32df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:54.170107 1890 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.190317 2887 raft_consensus.cc:1050] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:54.198613 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e017d73273c4a83bcfdb4c95caa32df with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:37:54.193114 2887 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:54.211663 2902 tablet_replica.cc:330] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:37:54.212518 2071 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:54.212947 2902 raft_consensus.cc:2229] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:54.216223 2902 raft_consensus.cc:2258] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:54.219087 2688 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.222800 2688 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.224622 2854 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:54.226291 2902 ts_tablet_manager.cc:1905] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:54.229972 2855 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:54.233639 1995 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.235575 2853 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.237944 2686 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.241904 2892 raft_consensus.cc:1050] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:54.243432 2139 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.244501 2854 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:54.244910 2904 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:54.246273 2892 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:54.247512 2071 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.248811 2823 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260927 19:37:54.257491 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d811e58acb4b4a3aa93ab155fd5ab1c2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:54.254350 1995 raft_consensus.cc:2932] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.281311 1995 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.291069 2788 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:54.300226 2142 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.300338 2887 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.302044 2892 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:54.306229 2141 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.307274 2686 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:54.309508 2854 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:54.311264 1893 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.348495 2071 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.349774 2686 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.355538 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet d811e58acb4b4a3aa93ab155fd5ab1c2 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:37:54.388849 1891 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.399240 2902 ts_tablet_manager.cc:1918] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:54.407384 2902 log.cc:1196] T 9e017d73273c4a83bcfdb4c95caa32df P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/9e017d73273c4a83bcfdb4c95caa32df
W20260927 19:37:54.420320 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 45980us
I20260927 19:37:54.454348 2902 tablet_replica.cc:330] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:37:54.454478 2910 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.455869 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet 9e017d73273c4a83bcfdb4c95caa32df (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:54.465543 1992 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.471922 1996 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:54.495790 2686 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.506870 2142 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.508487 2823 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:54.518318 2902 raft_consensus.cc:2229] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:54.519253 2902 raft_consensus.cc:2258] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:54.519944 1893 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.524842 2686 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.524639 2823 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.528182 2855 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:54.532260 2140 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.533916 2902 ts_tablet_manager.cc:1905] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:54.549994 2902 ts_tablet_manager.cc:1918] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:54.550535 2902 log.cc:1196] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/d811e58acb4b4a3aa93ab155fd5ab1c2
I20260927 19:37:54.552199 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet d811e58acb4b4a3aa93ab155fd5ab1c2 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:54.552484 2686 raft_consensus.cc:2932] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.556046 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ac63cb1bcdd141bb9de9fcb5d0f102af with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:54.564962 1890 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:54.591316 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet ac63cb1bcdd141bb9de9fcb5d0f102af with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:37:54.594413 2919 tablet_replica.cc:330] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:37:54.595563 2919 raft_consensus.cc:2229] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:54.598006 2141 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.601953 2919 raft_consensus.cc:2258] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:54.610082 2919 ts_tablet_manager.cc:1905] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:54.614982 2063 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:54.620954 2686 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.624518 2140 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:54.625886 2892 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:54.626856 2854 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:54.627404 2919 ts_tablet_manager.cc:1918] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:54.627842 2919 log.cc:1196] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/ac63cb1bcdd141bb9de9fcb5d0f102af
I20260927 19:37:54.630261 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet ac63cb1bcdd141bb9de9fcb5d0f102af (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:54.634760 2910 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.639613 2140 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.642567 2686 raft_consensus.cc:2932] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.649904 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4fe684e9563d491b9d4b3b99204450c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:54.657313 1890 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:54.668151 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fe684e9563d491b9d4b3b99204450c8 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:37:54.671514 2920 tablet_replica.cc:330] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:37:54.672675 2920 raft_consensus.cc:2229] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:54.673521 2920 raft_consensus.cc:2258] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:54.676515 2920 ts_tablet_manager.cc:1905] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:54.688712 2920 ts_tablet_manager.cc:1918] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:54.689248 2920 log.cc:1196] T 4fe684e9563d491b9d4b3b99204450c8 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/4fe684e9563d491b9d4b3b99204450c8
I20260927 19:37:54.691231 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 4fe684e9563d491b9d4b3b99204450c8 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:55.724961 1996 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:55.731428 2141 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:55.732105 2071 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:55.733147 1959 consensus_peers.cc:487] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4214caa28574754b61aad89471c77b9. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:55.734500 2823 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:55.735993 2823 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:55.742415 2923 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.744010 2141 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.751869 2071 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.758396 2063 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:55.763728 1893 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:55.772037 2141 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:55.773072 2034 consensus_peers.cc:487] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 876ffc23594245239e2688f5992774cf. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:55.773675 2854 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:55.782272 1992 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:55.785207 2892 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:55.828645 2854 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.836105 1992 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.837344 2142 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.846915 2139 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:55.852703 1892 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.874414 2063 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:55.875159 2108 consensus_peers.cc:487] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3acf4c2d78fb41ad974812ebf41d0478. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:55.874432 1992 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:55.875793 2893 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:55.877503 2893 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:55.885612 2913 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.889751 2071 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.912122 1992 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:55.919191 1995 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.921067 1893 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
W20260927 19:37:55.931664 1959 consensus_peers.cc:487] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6ecc556dac3458bb7b5ffda19a96158. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:55.935909 2071 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:55.936426 2139 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:55.937302 2921 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:55.938279 2855 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:55.948426 2921 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.950481 2068 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.953405 2139 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:55.977586 1893 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:56.252072 2943 ts_tablet_manager.cc:927] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:56.254494 2943 tablet_copy_client.cc:316] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:56.258641 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet a4214caa28574754b61aad89471c77b9 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:56.259198 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet a4214caa28574754b61aad89471c77b9 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-a4214caa28574754b61aad89471c77b9
I20260927 19:37:56.264683 2945 ts_tablet_manager.cc:927] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:56.270380 2006 tablet_copy_source_session.cc:205] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:56.272961 2945 tablet_copy_client.cc:316] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:37:56.273689 2943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4214caa28574754b61aad89471c77b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:56.275502 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet 3acf4c2d78fb41ad974812ebf41d0478 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:37:56.276094 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet 3acf4c2d78fb41ad974812ebf41d0478 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-3acf4c2d78fb41ad974812ebf41d0478
I20260927 19:37:56.286969 2155 tablet_copy_source_session.cc:205] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:56.290472 2945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acf4c2d78fb41ad974812ebf41d0478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:56.302398 2943 tablet_copy_client.cc:791] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:56.303134 2943 tablet_copy_client.cc:655] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:56.310902 2943 tablet_copy_client.cc:531] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:56.317500 2947 ts_tablet_manager.cc:927] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:56.321210 2947 tablet_copy_client.cc:316] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:56.323742 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet 876ffc23594245239e2688f5992774cf from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:56.324281 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet 876ffc23594245239e2688f5992774cf from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-876ffc23594245239e2688f5992774cf
I20260927 19:37:56.329487 2945 tablet_copy_client.cc:791] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:56.330073 2945 tablet_copy_client.cc:655] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:56.331459 2943 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:56.335379 2081 tablet_copy_source_session.cc:205] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:56.337039 2945 tablet_copy_client.cc:531] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:56.345525 2945 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:56.350805 2947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876ffc23594245239e2688f5992774cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:56.354241 2943 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:56.355383 2943 tablet_bootstrap.cc:492] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:56.356141 2943 ts_tablet_manager.cc:1397] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.004s
I20260927 19:37:56.358990 2943 raft_consensus.cc:348] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.359992 2943 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:56.360651 2943 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.374790 2943 ts_tablet_manager.cc:1428] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.018s user 0.011s sys 0.000s
I20260927 19:37:56.377590 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-a4214caa28574754b61aad89471c77b9 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:56.378261 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-a4214caa28574754b61aad89471c77b9 on tablet a4214caa28574754b61aad89471c77b9 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:56.385869 2947 tablet_copy_client.cc:791] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:56.386858 2947 tablet_copy_client.cc:655] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:56.390220 2947 tablet_copy_client.cc:531] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:56.399441 2947 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:56.451924 2945 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:56.453239 2945 tablet_bootstrap.cc:492] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:56.456202 2945 ts_tablet_manager.cc:1397] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.111s user 0.031s sys 0.011s
I20260927 19:37:56.467068 2947 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:56.468111 2947 tablet_bootstrap.cc:492] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:56.468840 2947 ts_tablet_manager.cc:1397] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.070s user 0.034s sys 0.023s
I20260927 19:37:56.469870 2945 raft_consensus.cc:348] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.470911 2945 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:56.471689 2947 raft_consensus.cc:348] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.472685 2947 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:56.471626 2945 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.473299 2947 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.477339 2945 ts_tablet_manager.cc:1428] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260927 19:37:56.479058 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-3acf4c2d78fb41ad974812ebf41d0478 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:37:56.479444 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-3acf4c2d78fb41ad974812ebf41d0478 on tablet 3acf4c2d78fb41ad974812ebf41d0478 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:56.479982 2947 ts_tablet_manager.cc:1428] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260927 19:37:56.481890 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-876ffc23594245239e2688f5992774cf received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:56.482266 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-876ffc23594245239e2688f5992774cf on tablet 876ffc23594245239e2688f5992774cf with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:56.520439 2947 ts_tablet_manager.cc:927] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:56.523262 2947 tablet_copy_client.cc:316] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:56.524690 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet f6ecc556dac3458bb7b5ffda19a96158 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:56.525149 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet f6ecc556dac3458bb7b5ffda19a96158 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-f6ecc556dac3458bb7b5ffda19a96158
I20260927 19:37:56.532121 2006 tablet_copy_source_session.cc:205] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:56.534911 2947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecc556dac3458bb7b5ffda19a96158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:56.550209 2947 tablet_copy_client.cc:791] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:56.550882 2947 tablet_copy_client.cc:655] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:56.553871 2947 tablet_copy_client.cc:531] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:56.560854 2947 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:56.580821 2947 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:56.581673 2947 tablet_bootstrap.cc:492] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:56.582298 2947 ts_tablet_manager.cc:1397] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.004s
I20260927 19:37:56.584600 2947 raft_consensus.cc:348] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.585500 2947 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:56.586517 2947 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:56.589208 2947 ts_tablet_manager.cc:1428] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 19:37:56.591089 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-f6ecc556dac3458bb7b5ffda19a96158 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:56.591480 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-f6ecc556dac3458bb7b5ffda19a96158 on tablet f6ecc556dac3458bb7b5ffda19a96158 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:56.788280 2688 raft_consensus.cc:1203] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:56.827543 2688 raft_consensus.cc:1203] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:56.865154 2688 raft_consensus.cc:1203] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:57.109566 2688 raft_consensus.cc:1203] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:37:57.330210 2923 raft_consensus.cc:1050] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:57.332588 2923 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.338385 2142 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.338742 2688 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.339862 2921 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:57.342832 2855 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:57.345737 2071 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.349949 2855 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:57.363638 2921 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.371551 2071 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.377110 2142 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.385538 2688 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.395604 1892 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:57.426671 2956 raft_consensus.cc:1050] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:57.425421 1995 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.429101 2956 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.436208 2068 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:57.436488 2688 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:57.438231 2921 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:57.440403 2923 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:57.447986 2938 raft_consensus.cc:1050] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:57.450075 2688 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.449584 2921 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.450582 2938 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.451867 2892 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:57.456019 2140 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.456856 2686 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.466434 1995 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.466305 2892 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:57.472158 2892 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:57.479012 2071 raft_consensus.cc:2932] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.481174 2068 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.525591 2967 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260927 19:37:57.536538 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 49032.2us
I20260927 19:37:57.538647 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a4214caa28574754b61aad89471c77b9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:57.550452 2687 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.554952 2938 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:57.563735 1893 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.574324 2892 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.595043 1996 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.600005 1996 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.602005 2855 raft_consensus.cc:1050] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:37:57.604142 2978 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:57.610536 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet a4214caa28574754b61aad89471c77b9 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
W20260927 19:37:57.665076 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48700.8us
W20260927 19:37:57.666960 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52208.2us
I20260927 19:37:57.605602 2855 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.669945 2140 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.671850 1891 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:57.685011 2688 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.695077 2984 tablet_replica.cc:330] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:37:57.696127 2984 raft_consensus.cc:2229] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:57.696856 2984 raft_consensus.cc:2258] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:57.711438 2068 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.712527 2984 ts_tablet_manager.cc:1905] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:57.740464 2687 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.746925 2921 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:37:57.749749 2140 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:57.758323 2956 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260927 19:37:57.760576 2983 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260927 19:37:57.762799 2686 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:57.762766 2984 ts_tablet_manager.cc:1918] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:57.764971 2984 log.cc:1196] T a4214caa28574754b61aad89471c77b9 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/a4214caa28574754b61aad89471c77b9
I20260927 19:37:57.764622 2982 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:57.766597 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet a4214caa28574754b61aad89471c77b9 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:57.772070 2974 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.775435 2063 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.776147 2956 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.778276 2139 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.806367 2067 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.808192 1996 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.812611 2921 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:37:57.814656 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 876ffc23594245239e2688f5992774cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:57.783684 2686 raft_consensus.cc:2932] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.821548 2688 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.822606 2142 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:37:57.832000 1893 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:37:57.845732 1892 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.851006 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 876ffc23594245239e2688f5992774cf with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:37:57.858379 2989 tablet_replica.cc:330] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:37:57.858836 2979 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.882287 2142 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:57.883245 2686 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.870404 2064 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.899144 1996 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:57.899274 1892 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.901542 2980 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260927 19:37:57.904560 2973 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20260927 19:37:57.904984 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 54297.4us
I20260927 19:37:57.906252 2686 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:57.907944 2971 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:57.915198 2980 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.917057 1996 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.928882 2140 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.933602 2688 raft_consensus.cc:2932] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.941119 2989 raft_consensus.cc:2229] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:57.941879 2989 raft_consensus.cc:2258] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:57.944197 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3acf4c2d78fb41ad974812ebf41d0478 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:57.961055 2989 ts_tablet_manager.cc:1905] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:57.983390 1891 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:57.993305 2989 ts_tablet_manager.cc:1918] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:57.994026 2989 log.cc:1196] T 876ffc23594245239e2688f5992774cf P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/876ffc23594245239e2688f5992774cf
I20260927 19:37:57.995229 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acf4c2d78fb41ad974812ebf41d0478 with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:37:57.999421 2991 tablet_replica.cc:330] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:37:58.000706 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 876ffc23594245239e2688f5992774cf (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:58.001731 2991 raft_consensus.cc:2229] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:58.002534 2991 raft_consensus.cc:2258] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:58.002446 1994 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:37:58.006616 2991 ts_tablet_manager.cc:1905] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:58.008067 2064 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:58.008109 2688 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:37:58.009716 2973 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:37:58.010605 2921 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:37:58.018508 2958 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:58.023458 2064 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:58.025516 2991 ts_tablet_manager.cc:1918] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:58.026052 2991 log.cc:1196] T 3acf4c2d78fb41ad974812ebf41d0478 P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/3acf4c2d78fb41ad974812ebf41d0478
I20260927 19:37:58.028280 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet 3acf4c2d78fb41ad974812ebf41d0478 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:58.031351 2688 raft_consensus.cc:2932] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:58.050242 1892 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:37:58.054085 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f6ecc556dac3458bb7b5ffda19a96158 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:37:58.067646 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ecc556dac3458bb7b5ffda19a96158 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:37:58.068580 2984 tablet_replica.cc:330] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:37:58.069821 2984 raft_consensus.cc:2229] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:37:58.070551 2984 raft_consensus.cc:2258] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:37:58.084537 2984 ts_tablet_manager.cc:1905] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:37:58.100739 2984 ts_tablet_manager.cc:1918] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:37:58.101202 2984 log.cc:1196] T f6ecc556dac3458bb7b5ffda19a96158 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/f6ecc556dac3458bb7b5ffda19a96158
I20260927 19:37:58.102766 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet f6ecc556dac3458bb7b5ffda19a96158 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:37:59.133713 1996 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.138996 2140 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:59.139307 1959 consensus_peers.cc:487] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ec895e575154c488c4fb6666ae75ce3. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:59.140363 2068 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:59.140242 2973 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:59.141542 2921 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:59.149554 2973 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.152175 2140 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.152175 2068 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.162869 2142 consensus_queue.cc:237] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.169678 2068 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:59.170598 1996 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:59.171324 2941 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:59.173034 2971 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:59.178968 1892 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260927 19:37:59.185708 2108 consensus_peers.cc:487] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba36610baf2046bc9b0bd15df9129f16. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:59.201885 2997 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.205242 2064 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.229983 1996 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.239742 2140 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.245625 1893 catalog_manager.cc:5495] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:59.256431 2064 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:37:59.259043 2108 consensus_peers.cc:487] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0017d74ca734f828f8096d7d474a209. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:59.259639 1996 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:59.262704 2941 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:59.267800 2941 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:59.276789 2971 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.283314 2068 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.288523 1994 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.297000 2068 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:37:59.304877 2034 consensus_peers.cc:487] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8814088a5337420ab29ac2b4b4e0f314. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:37:59.305104 1892 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.306746 2140 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:59.311728 1996 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:37:59.311585 2892 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:37:59.320839 2892 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:37:59.329216 2956 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.335605 1995 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.346655 2140 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:37:59.352092 1891 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:37:59.547461 3013 ts_tablet_manager.cc:927] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:59.550251 3013 tablet_copy_client.cc:316] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:37:59.552103 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet ba36610baf2046bc9b0bd15df9129f16 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:37:59.552611 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet ba36610baf2046bc9b0bd15df9129f16 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-ba36610baf2046bc9b0bd15df9129f16
I20260927 19:37:59.559557 2155 tablet_copy_source_session.cc:205] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:59.562947 3013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba36610baf2046bc9b0bd15df9129f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:59.578333 3013 tablet_copy_client.cc:791] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:59.578974 3013 tablet_copy_client.cc:655] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:59.584327 3013 tablet_copy_client.cc:531] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:59.593145 3013 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:59.605669 3015 ts_tablet_manager.cc:927] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:37:59.607694 3015 tablet_copy_client.cc:316] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:37:59.609072 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet 6ec895e575154c488c4fb6666ae75ce3 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:37:59.609624 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet 6ec895e575154c488c4fb6666ae75ce3 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-6ec895e575154c488c4fb6666ae75ce3
I20260927 19:37:59.612347 3013 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:59.613240 3013 tablet_bootstrap.cc:492] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:59.613891 3013 ts_tablet_manager.cc:1397] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.015s
I20260927 19:37:59.616088 3013 raft_consensus.cc:348] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.616994 3013 raft_consensus.cc:729] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:59.617573 2006 tablet_copy_source_session.cc:205] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:59.617636 3013 consensus_queue.cc:260] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.622452 3015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ec895e575154c488c4fb6666ae75ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:59.622659 3013 ts_tablet_manager.cc:1428] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260927 19:37:59.625002 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-ba36610baf2046bc9b0bd15df9129f16 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:37:59.625504 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-ba36610baf2046bc9b0bd15df9129f16 on tablet ba36610baf2046bc9b0bd15df9129f16 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:59.639611 3015 tablet_copy_client.cc:791] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:59.640218 3015 tablet_copy_client.cc:655] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:59.647001 3015 tablet_copy_client.cc:531] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:59.656138 3015 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:59.674489 3015 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:59.675376 3015 tablet_bootstrap.cc:492] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:59.676092 3015 ts_tablet_manager.cc:1397] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.008s
I20260927 19:37:59.678426 3015 raft_consensus.cc:348] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.679045 3015 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:59.680017 3015 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.681905 3015 ts_tablet_manager.cc:1428] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 19:37:59.683681 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-6ec895e575154c488c4fb6666ae75ce3 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:37:59.684043 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-6ec895e575154c488c4fb6666ae75ce3 on tablet 6ec895e575154c488c4fb6666ae75ce3 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:59.791208 3015 ts_tablet_manager.cc:927] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:37:59.793093 3015 tablet_copy_client.cc:316] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:37:59.794678 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet 8814088a5337420ab29ac2b4b4e0f314 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:37:59.795120 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet 8814088a5337420ab29ac2b4b4e0f314 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-8814088a5337420ab29ac2b4b4e0f314
I20260927 19:37:59.801035 2081 tablet_copy_source_session.cc:205] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:59.804206 3015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8814088a5337420ab29ac2b4b4e0f314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:59.810606 3013 ts_tablet_manager.cc:927] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:37:59.812740 3013 tablet_copy_client.cc:316] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:37:59.815502 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet c0017d74ca734f828f8096d7d474a209 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:37:59.816018 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet c0017d74ca734f828f8096d7d474a209 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-c0017d74ca734f828f8096d7d474a209
I20260927 19:37:59.827201 2155 tablet_copy_source_session.cc:205] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:37:59.827564 3015 tablet_copy_client.cc:791] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:59.828227 3015 tablet_copy_client.cc:655] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:59.830389 3013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0017d74ca734f828f8096d7d474a209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:37:59.834127 3015 tablet_copy_client.cc:531] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:59.843840 3013 tablet_copy_client.cc:791] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:37:59.844450 3013 tablet_copy_client.cc:655] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:37:59.844691 3015 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:59.854192 3013 tablet_copy_client.cc:531] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:37:59.866221 3015 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:59.866633 3013 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:37:59.867257 3015 tablet_bootstrap.cc:492] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:59.868050 3015 ts_tablet_manager.cc:1397] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20260927 19:37:59.870949 3015 raft_consensus.cc:348] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.871943 3015 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:59.872653 3015 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.875674 3015 ts_tablet_manager.cc:1428] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260927 19:37:59.887567 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-8814088a5337420ab29ac2b4b4e0f314 received from {username='slave'} at 127.0.0.1:60518
I20260927 19:37:59.887974 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-8814088a5337420ab29ac2b4b4e0f314 on tablet 8814088a5337420ab29ac2b4b4e0f314 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:37:59.900074 3013 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:37:59.900936 3013 tablet_bootstrap.cc:492] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:37:59.901664 3013 ts_tablet_manager.cc:1397] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.001s
I20260927 19:37:59.904371 3013 raft_consensus.cc:348] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.905180 3013 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:37:59.905786 3013 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:37:59.921192 3013 ts_tablet_manager.cc:1428] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260927 19:37:59.924098 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-c0017d74ca734f828f8096d7d474a209 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:37:59.924577 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-c0017d74ca734f828f8096d7d474a209 on tablet c0017d74ca734f828f8096d7d474a209 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:00.008080 2688 raft_consensus.cc:1203] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:00.168295 2688 raft_consensus.cc:1203] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:00.311928 2688 raft_consensus.cc:1203] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:00.328990 2686 raft_consensus.cc:1203] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:00.553435 2941 raft_consensus.cc:1050] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:00.555547 2941 consensus_queue.cc:237] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:00.561633 2686 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.561900 2068 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.562184 1995 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.563361 3007 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.565124 3007 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.567026 2971 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.580572 3007 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.591833 1996 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.597262 2068 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.604458 1893 catalog_manager.cc:5495] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:00.609753 2686 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.633692 2973 raft_consensus.cc:1050] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:00.635782 2973 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:00.645937 2686 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.647822 2921 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.649557 2142 consensus_queue.cc:237] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:00.655369 2688 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:00.655747 2068 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:00.657838 3007 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:00.658366 2142 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.659708 2971 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:00.660491 2921 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.687844 2063 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.688802 3025 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.689796 2973 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.692701 2063 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.709234 2921 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.716387 2068 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.722890 2686 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.731600 2687 raft_consensus.cc:2932] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.735325 2892 raft_consensus.cc:1050] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:00.739264 1890 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:00.741925 2892 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:00.731629 1891 catalog_manager.cc:5495] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.767354 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet ba36610baf2046bc9b0bd15df9129f16 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:38:00.796535 2686 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.797819 2892 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.798592 1996 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.799885 3026 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.811683 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ba36610baf2046bc9b0bd15df9129f16 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:00.815742 2142 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.818789 3012 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.825845 3045 tablet_replica.cc:330] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:38:00.825031 3012 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.827117 3045 raft_consensus.cc:2229] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:00.827828 3045 raft_consensus.cc:2258] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:00.834331 3045 ts_tablet_manager.cc:1905] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:00.833729 1996 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.835152 2941 raft_consensus.cc:1050] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:00.841748 2941 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:00.845299 2140 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.862133 2687 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.864785 2063 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.868980 2971 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:00.871959 3007 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:00.875365 1995 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:00.876812 3045 ts_tablet_manager.cc:1918] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:00.877385 3045 log.cc:1196] T ba36610baf2046bc9b0bd15df9129f16 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/ba36610baf2046bc9b0bd15df9129f16
I20260927 19:38:00.880908 3025 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:00.885310 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet ba36610baf2046bc9b0bd15df9129f16 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:00.894096 2686 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.929519 1891 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.970923 3007 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.975818 1995 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:00.985219 2143 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:00.986207 2063 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:00.987828 2921 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:00.988811 2685 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:00.997709 3040 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:01.001425 3046 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.006775 2687 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.004729 2143 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.015725 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6ec895e575154c488c4fb6666ae75ce3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:01.006050 2139 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.019824 1995 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.020337 2686 raft_consensus.cc:2932] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.028224 1890 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:01.054673 1893 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:01.065908 2068 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:01.073663 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ec895e575154c488c4fb6666ae75ce3 with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:38:01.080230 2686 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:01.081002 2142 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:01.082374 3012 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:01.084019 3026 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:01.094810 2140 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:01.095098 2892 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.096454 2142 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.101804 2686 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:01.102892 1996 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:01.103750 3025 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:01.105839 3007 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:01.117679 2685 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.115474 3025 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.123912 3054 tablet_replica.cc:330] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:38:01.126307 3054 raft_consensus.cc:2229] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:01.131878 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c0017d74ca734f828f8096d7d474a209 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:01.132828 3054 raft_consensus.cc:2258] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:01.131124 1995 raft_consensus.cc:2932] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.147693 3054 ts_tablet_manager.cc:1905] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:01.138244 2687 raft_consensus.cc:2932] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.155005 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8814088a5337420ab29ac2b4b4e0f314 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:01.165179 1893 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:01.166263 1890 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:01.182833 3054 ts_tablet_manager.cc:1918] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:01.183370 3054 log.cc:1196] T 6ec895e575154c488c4fb6666ae75ce3 P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/6ec895e575154c488c4fb6666ae75ce3
I20260927 19:38:01.185114 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet 6ec895e575154c488c4fb6666ae75ce3 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:01.189299 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet c0017d74ca734f828f8096d7d474a209 with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:38:01.193050 3054 tablet_replica.cc:330] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:38:01.189337 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 8814088a5337420ab29ac2b4b4e0f314 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:38:01.195492 3045 tablet_replica.cc:330] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:38:01.196416 3045 raft_consensus.cc:2229] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:01.197330 3045 raft_consensus.cc:2258] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:01.195495 3054 raft_consensus.cc:2229] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:01.198734 3054 raft_consensus.cc:2258] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:01.201519 3045 ts_tablet_manager.cc:1905] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:01.201946 3054 ts_tablet_manager.cc:1905] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:01.221777 3054 ts_tablet_manager.cc:1918] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:01.222178 3054 log.cc:1196] T c0017d74ca734f828f8096d7d474a209 P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/c0017d74ca734f828f8096d7d474a209
I20260927 19:38:01.223896 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet c0017d74ca734f828f8096d7d474a209 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:01.236056 3045 ts_tablet_manager.cc:1918] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:01.236469 3045 log.cc:1196] T 8814088a5337420ab29ac2b4b4e0f314 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/8814088a5337420ab29ac2b4b4e0f314
I20260927 19:38:01.237810 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 8814088a5337420ab29ac2b4b4e0f314 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:02.263056 1995 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:38:02.270159 1959 consensus_peers.cc:487] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c4fa5fe3da244bb9e845557555810b8. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:02.270227 2140 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:02.273205 3046 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:02.276235 2068 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:02.280802 3020 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:02.287339 3020 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.294085 2063 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.294085 2140 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.300922 1995 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.308174 1892 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:02.316746 2063 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:38:02.318868 1959 consensus_peers.cc:487] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6886530f9f741c2acaff96949f65f19. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:02.321406 2140 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:02.332652 3020 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260927 19:38:02.337404 3065 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260927 19:38:02.361799 3066 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.364545 2064 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.370832 2143 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.386147 2139 consensus_queue.cc:237] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.393857 2063 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:38:02.395058 2108 consensus_peers.cc:487] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e0ad2bfdf76425782a9548c96da6ca7. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:02.395166 3025 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:02.427873 1995 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:02.454133 3007 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260927 19:38:02.580936 3057 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.589690 1891 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.632714 1995 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.637043 2063 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.646684 2139 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:02.647490 2064 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.651600 3012 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260927 19:38:02.655285 2034 consensus_peers.cc:487] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 29707548f67846318643f37690b25103. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:02.661523 1995 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:02.662235 1892 catalog_manager.cc:5495] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.671973 3012 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:02.688861 3080 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.698665 2139 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.699285 1996 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:02.713711 1893 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:02.751995 3085 ts_tablet_manager.cc:927] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:02.761003 3085 tablet_copy_client.cc:316] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:38:02.763531 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet 3c4fa5fe3da244bb9e845557555810b8 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:38:02.764254 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet 3c4fa5fe3da244bb9e845557555810b8 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-3c4fa5fe3da244bb9e845557555810b8
I20260927 19:38:02.780076 2006 tablet_copy_source_session.cc:205] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:02.784648 3085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fa5fe3da244bb9e845557555810b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:02.802313 3085 tablet_copy_client.cc:791] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:02.803119 3085 tablet_copy_client.cc:655] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:02.807646 3085 tablet_copy_client.cc:531] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:02.828238 3085 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:02.854013 3085 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:02.855150 3085 tablet_bootstrap.cc:492] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:02.856040 3085 ts_tablet_manager.cc:1397] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260927 19:38:02.860073 3085 raft_consensus.cc:348] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.861305 3085 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:02.862154 3085 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.873055 3088 ts_tablet_manager.cc:927] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:02.874744 3085 ts_tablet_manager.cc:1428] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.018s user 0.011s sys 0.009s
I20260927 19:38:02.877713 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-3c4fa5fe3da244bb9e845557555810b8 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:38:02.878211 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-3c4fa5fe3da244bb9e845557555810b8 on tablet 3c4fa5fe3da244bb9e845557555810b8 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:02.881413 3088 tablet_copy_client.cc:316] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:38:02.889153 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet c6886530f9f741c2acaff96949f65f19 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:38:02.889752 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet c6886530f9f741c2acaff96949f65f19 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-c6886530f9f741c2acaff96949f65f19
I20260927 19:38:02.899875 3085 ts_tablet_manager.cc:927] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:02.901449 2006 tablet_copy_source_session.cc:205] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:02.906677 3088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6886530f9f741c2acaff96949f65f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:02.914637 3085 tablet_copy_client.cc:316] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:38:02.916425 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet 9e0ad2bfdf76425782a9548c96da6ca7 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:38:02.916937 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet 9e0ad2bfdf76425782a9548c96da6ca7 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-9e0ad2bfdf76425782a9548c96da6ca7
I20260927 19:38:02.924935 2155 tablet_copy_source_session.cc:205] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:02.929090 3088 tablet_copy_client.cc:791] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:02.929817 3088 tablet_copy_client.cc:655] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:02.932557 3085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ad2bfdf76425782a9548c96da6ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:02.938943 3088 tablet_copy_client.cc:531] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:02.949909 3088 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:02.950776 3085 tablet_copy_client.cc:791] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:02.951462 3085 tablet_copy_client.cc:655] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:02.958770 3085 tablet_copy_client.cc:531] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:02.976269 3088 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:02.977417 3088 tablet_bootstrap.cc:492] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:02.978343 3088 ts_tablet_manager.cc:1397] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.008s
I20260927 19:38:02.979135 3085 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:02.981402 3088 raft_consensus.cc:348] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.982486 3088 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:02.983206 3088 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:02.988467 3088 ts_tablet_manager.cc:1428] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260927 19:38:02.993316 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-c6886530f9f741c2acaff96949f65f19 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:38:02.994022 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-c6886530f9f741c2acaff96949f65f19 on tablet c6886530f9f741c2acaff96949f65f19 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:03.008926 3085 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:03.011088 3085 tablet_bootstrap.cc:492] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:03.011842 3085 ts_tablet_manager.cc:1397] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.000s
I20260927 19:38:03.014480 3085 raft_consensus.cc:348] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:03.015363 3085 raft_consensus.cc:729] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:03.016019 3085 consensus_queue.cc:260] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:03.018242 3085 ts_tablet_manager.cc:1428] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 19:38:03.020493 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-9e0ad2bfdf76425782a9548c96da6ca7 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:38:03.020937 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-9e0ad2bfdf76425782a9548c96da6ca7 on tablet 9e0ad2bfdf76425782a9548c96da6ca7 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:03.073881 3085 ts_tablet_manager.cc:927] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:03.076746 3085 tablet_copy_client.cc:316] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:38:03.078531 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet 29707548f67846318643f37690b25103 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
I20260927 19:38:03.079109 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet 29707548f67846318643f37690b25103 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-29707548f67846318643f37690b25103
I20260927 19:38:03.087709 2081 tablet_copy_source_session.cc:205] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:03.090912 3085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29707548f67846318643f37690b25103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:03.107190 3085 tablet_copy_client.cc:791] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:03.107797 3085 tablet_copy_client.cc:655] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:03.111976 3085 tablet_copy_client.cc:531] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:03.121762 3085 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:03.167421 3085 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:03.168226 3085 tablet_bootstrap.cc:492] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:03.168778 3085 ts_tablet_manager.cc:1397] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.007s
I20260927 19:38:03.170809 3085 raft_consensus.cc:348] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:03.171572 3085 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:03.172130 3085 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:03.173923 3085 ts_tablet_manager.cc:1428] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260927 19:38:03.175807 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-29707548f67846318643f37690b25103 received from {username='slave'} at 127.0.0.1:60518
I20260927 19:38:03.176206 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-29707548f67846318643f37690b25103 on tablet 29707548f67846318643f37690b25103 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:03.245123 2687 raft_consensus.cc:1203] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:03.302203 2687 raft_consensus.cc:1203] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:03.495688 2687 raft_consensus.cc:1203] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:03.645422 2687 raft_consensus.cc:1203] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:03.791343 3098 raft_consensus.cc:1050] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:03.793622 3098 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:03.809223 2140 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.809434 2064 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.810845 3096 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.811080 2687 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.811679 3098 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.813086 3046 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:03.832171 2142 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:03.830296 3100 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:03.851424 2687 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:03.857292 2068 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:03.858692 1893 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:03.869999 3100 raft_consensus.cc:1050] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:03.874087 3100 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:03.908680 3078 raft_consensus.cc:1050] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:03.910723 2140 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.912034 2685 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.911473 3078 consensus_queue.cc:237] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:03.912273 3096 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:03.914899 3046 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.927481 2064 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.933112 1995 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.933451 2685 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.933992 3100 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.934659 3078 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.936831 3007 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.945648 2064 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:03.947237 3007 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:03.992825 3100 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.012929 2140 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.022358 1992 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:04.027644 2687 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.029493 3098 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:04.042451 3098 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.044970 2686 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.045056 2687 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.068929 2063 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.065040 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3c4fa5fe3da244bb9e845557555810b8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:04.071774 3046 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:04.074069 1890 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.082316 1891 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:04.103065 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4fa5fe3da244bb9e845557555810b8 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:04.106329 3057 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.115697 2063 raft_consensus.cc:2932] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.139739 3117 tablet_replica.cc:330] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:38:04.132903 1996 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:04.147356 2686 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.148203 3117 raft_consensus.cc:2229] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:04.149180 3117 raft_consensus.cc:2258] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:04.149811 3098 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:04.154265 2140 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.155480 3098 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:04.155326 3117 ts_tablet_manager.cc:1905] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:04.160007 3100 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.169060 2143 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.169628 2068 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.172946 2686 raft_consensus.cc:2932] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.182825 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c6886530f9f741c2acaff96949f65f19 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:04.196702 1890 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.211969 3117 ts_tablet_manager.cc:1918] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:04.211242 1892 catalog_manager.cc:5495] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:04.212428 3117 log.cc:1196] T 3c4fa5fe3da244bb9e845557555810b8 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/3c4fa5fe3da244bb9e845557555810b8
I20260927 19:38:04.215116 1996 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:38:04.218521 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 95907.4us
I20260927 19:38:04.220884 2686 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.221680 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet 3c4fa5fe3da244bb9e845557555810b8 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:04.240257 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet c6886530f9f741c2acaff96949f65f19 with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:38:04.240861 3080 raft_consensus.cc:1050] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:04.243532 3080 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:04.274633 2686 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:04.277063 3121 tablet_replica.cc:330] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: stopping tablet replica
I20260927 19:38:04.277391 3058 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:04.283269 3121 raft_consensus.cc:2229] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:04.287649 3121 pending_rounds.cc:62] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Trying to abort 1 pending ops.
I20260927 19:38:04.291718 1996 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:04.294410 2144 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:04.297914 3012 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:04.299259 3058 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:04.289536 3121 pending_rounds.cc:68] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7334043172336775168 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c6886530f9f741c2acaff96949f65f19" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } } }
I20260927 19:38:04.320997 3121 raft_consensus.cc:2894] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260927 19:38:04.321511 3121 raft_consensus.cc:2866] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260927 19:38:04.326529 3121 raft_consensus.cc:2258] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:04.332741 3012 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.336052 3121 ts_tablet_manager.cc:1905] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:04.339387 2686 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.343050 2140 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.351589 1891 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.354089 2144 consensus_queue.cc:237] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:04.364360 3121 ts_tablet_manager.cc:1918] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:04.366308 3121 log.cc:1196] T c6886530f9f741c2acaff96949f65f19 P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/c6886530f9f741c2acaff96949f65f19
I20260927 19:38:04.366567 2064 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.367826 2686 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.368315 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet c6886530f9f741c2acaff96949f65f19 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:04.371466 3007 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:04.372666 3078 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:04.377287 1995 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.387352 3120 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.393545 2686 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.399658 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9e0ad2bfdf76425782a9548c96da6ca7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:04.401502 2064 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:04.428727 2063 raft_consensus.cc:2932] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.433633 1893 catalog_manager.cc:5495] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:04.455927 1992 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.457744 3058 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:04.458599 2685 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:04.461256 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0ad2bfdf76425782a9548c96da6ca7 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:38:04.466482 3058 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:04.469413 3080 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.470674 1992 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.486711 2685 raft_consensus.cc:2932] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:04.504144 3128 tablet_replica.cc:330] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:38:04.512240 1892 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index 3 to 4, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 1 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:04.514644 1879 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 29707548f67846318643f37690b25103 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:04.527063 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet 29707548f67846318643f37690b25103 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:04.528107 3117 tablet_replica.cc:330] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:38:04.529275 3117 raft_consensus.cc:2229] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:04.530004 3117 raft_consensus.cc:2258] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:04.534006 3128 raft_consensus.cc:2229] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:04.534809 3128 raft_consensus.cc:2258] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:04.538853 3128 ts_tablet_manager.cc:1905] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:04.540409 3117 ts_tablet_manager.cc:1905] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:04.556696 3128 ts_tablet_manager.cc:1918] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:04.557073 3117 ts_tablet_manager.cc:1918] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:04.557205 3128 log.cc:1196] T 9e0ad2bfdf76425782a9548c96da6ca7 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/9e0ad2bfdf76425782a9548c96da6ca7
I20260927 19:38:04.557669 3117 log.cc:1196] T 29707548f67846318643f37690b25103 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/29707548f67846318643f37690b25103
I20260927 19:38:04.559332 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 9e0ad2bfdf76425782a9548c96da6ca7 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:04.559881 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet 29707548f67846318643f37690b25103 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:04.711308 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:04.778999 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:04.779757 1992 raft_consensus.cc:595] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:04.782673 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:04.783262 1992 raft_consensus.cc:595] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:04.785867 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:04.786320 1992 raft_consensus.cc:595] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:04.788597 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:04.789062 1992 raft_consensus.cc:595] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:04.815538 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:04.816144 2140 raft_consensus.cc:595] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:04.905642 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:04.906276 1992 raft_consensus.cc:595] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:04.907930 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:38:04.909693 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:04.910544 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:04.922554 3096 raft_consensus.cc:979] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:04.923100 3122 raft_consensus.cc:1067] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:04.925381 2686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:04.925966 2686 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:04.929708 2686 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:04.953604 2686 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:04.973465 2686 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
W20260927 19:38:05.070849 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 107469us
I20260927 19:38:05.110966 3133 raft_consensus.cc:979] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:05.111193 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:05.111541 3133 raft_consensus.cc:1067] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:05.111899 2140 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.114385 2686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:05.114938 2686 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:05.115245 2686 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.123220 2686 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.127770 2686 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:05.129856 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:05.130863 2144 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.141314 3078 raft_consensus.cc:979] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:05.142060 3078 raft_consensus.cc:1067] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:05.146381 2144 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.150754 2686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34940
I20260927 19:38:05.177304 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:05.178047 1995 raft_consensus.cc:3032] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:05.178236 2654 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:05.178390 1995 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:05.179440 1995 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.181010 1995 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.191747 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:05.192453 1992 raft_consensus.cc:3032] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:05.192728 1992 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:05.193413 1992 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.195439 1992 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.203544 3136 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:05.206377 1995 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.207036 1992 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.215379 3136 raft_consensus.cc:686] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:05.216572 3136 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:05.225077 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 46944.6us
I20260927 19:38:05.227790 2654 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:05.155593 2686 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:05.266376 3140 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:05.277873 3140 raft_consensus.cc:686] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:05.300004 3133 raft_consensus.cc:979] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:05.300695 3133 raft_consensus.cc:1067] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:05.277026 2686 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.288084 3140 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.323457 2140 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.365119 2686 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:05.368224 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 154829us
I20260927 19:38:05.368865 3141 raft_consensus.cc:979] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:05.369760 3098 raft_consensus.cc:1067] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:05.389195 2686 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:05.398871 3139 raft_consensus.cc:979] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:05.399623 3096 raft_consensus.cc:1067] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:05.401585 2686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:05.402146 2686 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:05.402464 2686 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.411281 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:05.411890 2068 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:05.412295 2068 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.414719 2686 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.419126 2068 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:05.420925 2068 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:05.422052 2686 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:05.429096 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 198251us
I20260927 19:38:05.440097 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:05.440925 2067 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.450098 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:05.450814 2068 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.451046 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:05.451808 1995 raft_consensus.cc:3032] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:05.452196 1995 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:05.453504 1995 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:05.454977 1995 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.458802 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:05.459406 2142 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.460282 2068 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.467612 2142 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:05.470409 1995 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:05.470839 2067 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.472875 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:05.475780 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:05.479956 1995 raft_consensus.cc:3032] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:05.480110 2068 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:05.480408 1995 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:05.481426 1995 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.476742 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:05.482965 1995 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.483170 2142 raft_consensus.cc:3032] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:05.486757 2142 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:05.487632 2142 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.483232 1892 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:05.489435 2142 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.490182 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:05.491016 3012 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:05.491464 3012 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:05.480718 2068 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.492173 3012 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:05.499965 2653 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:05.500881 3136 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:05.501320 2653 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:05.501493 3136 raft_consensus.cc:686] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:05.502466 3142 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:05.503046 3142 raft_consensus.cc:686] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:05.502430 3136 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.504047 3142 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:05.500571 2142 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.546070 1995 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:05.556488 1893 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:05.540738 2068 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:05.570649 2064 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:05.570910 2068 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:05.572932 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:05.573570 1995 raft_consensus.cc:3032] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:05.573870 1995 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:05.574608 1995 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:05.574865 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:05.575600 1995 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.575544 2142 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:05.586676 1995 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:05.588651 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:05.591825 3127 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:05.592336 3127 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:05.593286 3127 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:05.594178 2142 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:05.626222 1892 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:05.626812 1893 catalog_manager.cc:5495] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:05.627882 1892 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:05.629429 1892 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:05.686139 2142 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:05.687398 3142 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:05.704952 1995 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:05.743089 1996 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:05.744881 2140 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:38:05.746316 1959 consensus_peers.cc:487] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da05114b545f4ec9a25e86d9c059fea3. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:05.746527 3141 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:05.747841 3136 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:05.757686 2067 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:05.760635 3141 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:05.812634 3134 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:05.814703 2067 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:05.831326 2144 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:05.840734 1893 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:05.842739 2067 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 19:38:05.847219 2108 consensus_peers.cc:487] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f19d4799cf134d51b543a2952f564adc. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:05.851727 2142 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:05.854367 2139 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:05.856011 3142 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:05.861125 1995 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:05.865731 3142 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:05.861191 3132 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260927 19:38:05.875777 1996 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:05.877265 3132 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:05.887919 1992 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:05.891148 3080 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:05.900794 3145 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:05.904749 1996 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:05.940930 1995 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:38:06.044310 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 89810.2us
I20260927 19:38:06.056644 1891 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260927 19:38:06.072300 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 138031us
I20260927 19:38:06.087540 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:06.087759 2139 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:06.093987 1996 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260927 19:38:06.095355 1959 consensus_peers.cc:487] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fcc7a171b3e443d8f28fd70f8742470. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:06.096501 3142 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:06.097572 2064 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.101887 2139 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:06.104369 2068 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:06.104559 2139 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:06.106268 3098 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:06.106755 3142 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:06.107206 2068 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:06.122366 2144 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:06.128593 3140 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:06.128397 3134 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.128415 3142 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 19:38:06.136493 3080 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:06.140034 3127 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:06.139863 2139 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.159708 1995 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.165927 2067 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:06.174162 3098 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260927 19:38:06.191808 1959 consensus_peers.cc:487] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 228a1cf257434cbe9e5616dbd94e1864. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:06.194636 2140 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:06.198374 3133 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:06.244913 1992 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 19:38:06.273758 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08000da080 after lost signal to thread 821
I20260927 19:38:06.275154 3182 ts_tablet_manager.cc:927] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
W20260927 19:38:06.275360 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08000e8560 after lost signal to thread 1927
W20260927 19:38:06.277042 2017 debug-util.cc:398] Leaking SignalData structure 0x7b0800118dc0 after lost signal to thread 2020
W20260927 19:38:06.279861 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08000db860 after lost signal to thread 2095
W20260927 19:38:06.284616 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08002fc3c0 after lost signal to thread 2169
W20260927 19:38:06.285842 2017 debug-util.cc:398] Leaking SignalData structure 0x7b080023ab20 after lost signal to thread 2712
W20260927 19:38:06.286355 2017 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdf80 after lost signal to thread 3107
I20260927 19:38:06.295038 2067 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 19:38:06.299979 3176 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:06.302485 1892 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.307307 2064 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.309329 3182 tablet_copy_client.cc:316] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:38:06.312034 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet da05114b545f4ec9a25e86d9c059fea3 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:38:06.312633 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet da05114b545f4ec9a25e86d9c059fea3 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-da05114b545f4ec9a25e86d9c059fea3
I20260927 19:38:06.320611 3141 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.327528 3098 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:06.350358 2006 tablet_copy_source_session.cc:205] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:06.373651 3182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da05114b545f4ec9a25e86d9c059fea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:06.377683 2067 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.350947 2144 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:06.398947 3187 ts_tablet_manager.cc:927] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:06.420722 1893 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:06.443893 3187 tablet_copy_client.cc:316] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:38:06.446327 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet f19d4799cf134d51b543a2952f564adc from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:38:06.447407 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet f19d4799cf134d51b543a2952f564adc from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-f19d4799cf134d51b543a2952f564adc
I20260927 19:38:06.455856 2155 tablet_copy_source_session.cc:205] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:06.471733 3182 tablet_copy_client.cc:791] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:06.472553 3182 tablet_copy_client.cc:655] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:06.474234 3187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19d4799cf134d51b543a2952f564adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:06.476930 3182 tablet_copy_client.cc:531] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:06.482252 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:06.482851 1995 raft_consensus.cc:595] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:06.491652 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33febda9075249dc9de325af8884cce5"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:06.492239 1995 raft_consensus.cc:595] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:06.492624 3182 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:06.497414 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:06.498049 1995 raft_consensus.cc:595] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:06.516135 3187 tablet_copy_client.cc:791] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:06.521306 3187 tablet_copy_client.cc:655] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:06.527334 3187 tablet_copy_client.cc:531] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:06.546566 3187 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:06.555459 2067 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:06.556064 2067 raft_consensus.cc:595] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:06.558828 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:06.559551 1992 raft_consensus.cc:595] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:06.564401 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:06.565060 2144 raft_consensus.cc:595] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:06.571841 3132 raft_consensus.cc:979] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:06.572337 3132 raft_consensus.cc:1067] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:06.573294 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:06.573907 1992 raft_consensus.cc:595] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:06.575282 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260927 19:38:06.576093 2686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34940
I20260927 19:38:06.576632 2686 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:06.576856 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:06.576967 2686 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.577579 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:06.590435 2686 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:06.596923 3187 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:06.600157 3187 tablet_bootstrap.cc:492] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:06.601051 3187 ts_tablet_manager.cc:1397] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.017s
I20260927 19:38:06.601847 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:06.603708 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:06.604350 2144 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:06.604664 2144 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:06.605428 2144 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:06.606600 2144 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.613662 1992 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.618676 2686 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:06.623827 1992 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:06.618515 2144 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:06.625373 3187 raft_consensus.cc:348] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.626125 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:06.637531 3187 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:06.638226 3187 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.638778 3142 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:06.639472 3142 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:06.640810 3187 ts_tablet_manager.cc:1428] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20260927 19:38:06.640405 3142 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:06.643987 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-f19d4799cf134d51b543a2952f564adc received from {username='slave'} at 127.0.0.1:46930
I20260927 19:38:06.644434 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-f19d4799cf134d51b543a2952f564adc on tablet f19d4799cf134d51b543a2952f564adc with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:06.664882 3133 raft_consensus.cc:979] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:06.665397 3133 raft_consensus.cc:1067] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:06.669926 2064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:06.670362 3187 ts_tablet_manager.cc:927] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:06.670539 2064 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:06.670938 2064 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.636857 3182 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:06.678640 3182 tablet_bootstrap.cc:492] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:06.699752 1892 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:06.712034 2064 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.679924 3182 ts_tablet_manager.cc:1397] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.188s user 0.027s sys 0.023s
I20260927 19:38:06.720009 3187 tablet_copy_client.cc:316] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:38:06.721797 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet 0fcc7a171b3e443d8f28fd70f8742470 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:38:06.722424 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet 0fcc7a171b3e443d8f28fd70f8742470 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-0fcc7a171b3e443d8f28fd70f8742470
I20260927 19:38:06.722779 2687 raft_consensus.cc:1203] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:06.729753 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:06.730361 1996 raft_consensus.cc:3032] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:06.730643 1996 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:06.731343 1996 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.732429 1996 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.745437 2064 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:06.748483 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:06.749099 2144 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.753767 1996 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:06.731828 2006 tablet_copy_source_session.cc:205] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:06.756167 3138 raft_consensus.cc:979] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:06.756377 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:06.756664 3134 raft_consensus.cc:1067] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:06.759733 3187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcc7a171b3e443d8f28fd70f8742470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:06.760689 3175 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:06.761713 3175 raft_consensus.cc:686] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:06.762502 3175 consensus_queue.cc:237] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.765127 3134 raft_consensus.cc:979] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:06.773535 2140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33febda9075249dc9de325af8884cce5"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:38:06.776208 2140 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:06.776553 2140 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.779538 3182 raft_consensus.cc:348] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.780507 3182 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:06.781183 3182 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.775561 2144 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:06.787020 3182 ts_tablet_manager.cc:1428] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.069s user 0.004s sys 0.002s
I20260927 19:38:06.787901 3098 raft_consensus.cc:1067] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:06.793478 3196 ts_tablet_manager.cc:927] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:06.798683 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-da05114b545f4ec9a25e86d9c059fea3 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:38:06.788641 2140 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.804905 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-da05114b545f4ec9a25e86d9c059fea3 on tablet da05114b545f4ec9a25e86d9c059fea3 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:06.808157 2140 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:06.808579 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:06.809201 2068 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:06.809571 2068 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.809851 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:06.810539 1996 raft_consensus.cc:3032] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:06.810771 1996 raft_consensus.cc:729] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:06.812618 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:06.813236 2067 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.813601 1996 consensus_queue.cc:260] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.819689 1996 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.824719 2067 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:06.827003 2110 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:06.829921 3132 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:06.830928 3132 raft_consensus.cc:686] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:06.835364 1996 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:06.842202 2068 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.848984 2068 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:06.847623 3132 consensus_queue.cc:237] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.852715 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:06.853333 2140 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.853595 1892 catalog_manager.cc:5495] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:06.860203 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:06.863818 1992 raft_consensus.cc:3032] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:06.864457 1992 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:06.865433 1992 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.868100 3196 tablet_copy_client.cc:316] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.129:39941
I20260927 19:38:06.878778 2140 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:06.879949 1992 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.880968 2006 tablet_copy_service.cc:140] P 3d1bc2cd9efe43399ddba396f5ddff85: Received BeginTabletCopySession request for tablet 228a1cf257434cbe9e5616dbd94e1864 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:56618)
I20260927 19:38:06.882344 2037 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:06.881608 2006 tablet_copy_service.cc:161] P 3d1bc2cd9efe43399ddba396f5ddff85: Beginning new tablet copy session on tablet 228a1cf257434cbe9e5616dbd94e1864 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:56618: session id = 4e0335d1a85f421380029b95a91bf0c4-228a1cf257434cbe9e5616dbd94e1864
I20260927 19:38:06.883545 3176 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:06.884007 3176 raft_consensus.cc:686] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:06.884812 3176 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.931855 1992 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:06.936661 3187 tablet_copy_client.cc:791] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:06.937288 3187 tablet_copy_client.cc:655] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:06.941416 3187 tablet_copy_client.cc:531] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:06.943599 1892 catalog_manager.cc:5495] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:06.945461 3171 raft_consensus.cc:979] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: : Instructing follower 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:06.945955 3171 raft_consensus.cc:1067] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Signalling peer 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:06.947734 1996 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:60296
I20260927 19:38:06.948283 1996 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:06.948635 1996 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.959061 2006 tablet_copy_source_session.cc:205] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:06.965530 3196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228a1cf257434cbe9e5616dbd94e1864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:06.965641 3141 raft_consensus.cc:979] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:06.966140 3134 raft_consensus.cc:1067] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:06.966688 1996 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.967940 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:06.968525 2068 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:06.968861 2068 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.969547 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:06.970124 2140 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.977643 2140 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:06.978225 1996 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:06.977677 2068 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.982781 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:06.983384 1995 raft_consensus.cc:3032] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:06.983709 1995 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:06.984452 1995 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:06.986253 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:06.986814 2071 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:06.987056 2071 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:06.987738 2071 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:06.988708 2071 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.990907 1995 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:06.994683 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:06.997344 3098 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:06.997869 2071 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:06.999297 3187 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:06.999832 3098 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:07.000639 3098 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:07.002415 1995 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:07.005726 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:06.989107 1892 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:07.009059 3176 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:07.009877 3176 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:07.010875 2068 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:07.012696 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:07.013720 2142 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:07.020150 2142 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:07.024328 3176 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:07.040884 3196 tablet_copy_client.cc:791] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:07.043690 3196 tablet_copy_client.cc:655] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:07.051714 3196 tablet_copy_client.cc:531] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:07.051335 1891 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:07.050083 1892 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:07.078328 3196 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:07.079773 3201 raft_consensus.cc:979] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:07.081849 3134 raft_consensus.cc:1067] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:07.084033 2686 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:07.084633 2686 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:07.084878 2686 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:07.095911 2686 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:07.101164 2686 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:07.103410 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:07.104038 1996 raft_consensus.cc:3032] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:07.110338 1996 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:07.104302 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:07.115715 2140 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:07.120456 1996 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:07.125453 1996 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:07.129777 2140 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:07.130812 2654 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:07.137697 1996 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:07.136981 3142 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:07.139128 3142 raft_consensus.cc:686] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:07.140091 3142 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:07.180193 3187 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:07.181100 3187 tablet_bootstrap.cc:492] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:07.181843 3187 ts_tablet_manager.cc:1397] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.183s user 0.035s sys 0.031s
I20260927 19:38:07.188110 3187 raft_consensus.cc:348] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:07.190231 3187 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:07.189044 1891 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:07.190958 3187 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:07.193917 3187 ts_tablet_manager.cc:1428] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260927 19:38:07.196568 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-0fcc7a171b3e443d8f28fd70f8742470 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:38:07.197000 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-0fcc7a171b3e443d8f28fd70f8742470 on tablet 0fcc7a171b3e443d8f28fd70f8742470 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:07.204483 3120 raft_consensus.cc:1050] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:07.207267 3120 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:07.217523 2142 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.219473 3175 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:07.221506 2685 raft_consensus.cc:1203] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:07.227365 3196 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:07.228652 3196 tablet_bootstrap.cc:492] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:07.244570 2142 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:07.251199 3136 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260927 19:38:07.323962 3196 ts_tablet_manager.cc:1397] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 0.254s user 0.023s sys 0.002s
I20260927 19:38:07.519176 1992 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.521908 1992 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.524504 1992 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 19:38:07.481189 3196 raft_consensus.cc:348] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:07.529269 2139 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.581385 3196 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:07.550657 3221 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:07.552413 3176 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260927 19:38:07.553534 3127 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260927 19:38:07.582195 3196 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:07.561689 1991 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:07.531634 2685 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:07.585675 3145 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260927 19:38:07.600198 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44723.7us
I20260927 19:38:07.600888 1996 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:07.603601 2139 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.607151 2071 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:07.608537 2071 raft_consensus.cc:1261] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.617287 3142 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:07.621451 1993 raft_consensus.cc:1261] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:07.624203 3196 ts_tablet_manager.cc:1428] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.284s user 0.008s sys 0.008s
I20260927 19:38:07.641404 2006 tablet_copy_service.cc:342] P 3d1bc2cd9efe43399ddba396f5ddff85: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-228a1cf257434cbe9e5616dbd94e1864 received from {username='slave'} at 127.0.0.1:56618
I20260927 19:38:07.641987 2006 tablet_copy_service.cc:433] P 3d1bc2cd9efe43399ddba396f5ddff85: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-228a1cf257434cbe9e5616dbd94e1864 on tablet 228a1cf257434cbe9e5616dbd94e1864 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:07.649715 2139 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 19:38:07.653002 3012 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:07.655701 2687 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Advancing to term 2
I20260927 19:38:07.658551 3211 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260927 19:38:07.655545 3175 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260927 19:38:07.658408 1994 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:07.667672 2687 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 19:38:07.678056 3012 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:07.665774 3138 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:07.672667 3145 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20260927 19:38:07.718164 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 66403.4us
I20260927 19:38:07.763165 3218 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
I20260927 19:38:07.765772 3120 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:07.772466 3210 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
W20260927 19:38:07.822557 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 147957us
I20260927 19:38:07.827972 2686 raft_consensus.cc:1203] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260927 19:38:07.840806 2071 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 19:38:07.842751 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 93796.4us
I20260927 19:38:07.910274 3214 raft_consensus.cc:1050] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:07.913587 2141 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:07.939709 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:07.948248 3214 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:07.968981 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 97991.8us
I20260927 19:38:07.976168 3145 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:08.004169 3098 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:08.017151 3012 raft_consensus.cc:1050] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:08.030153 3219 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 19:38:08.050575 2687 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:38:08.125461 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 64433.9us
I20260927 19:38:08.131655 3012 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:08.193586 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 46166.7us
I20260927 19:38:08.173429 2684 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 19:38:08.207437 2687 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:08.208355 2066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:08.208953 2066 raft_consensus.cc:595] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:08.205741 2063 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:08.213629 1891 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:08.236179 2139 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:08.225801 3240 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260927 19:38:08.217514 3220 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260927 19:38:08.264752 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:08.265588 1996 raft_consensus.cc:595] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:08.283545 2142 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 19:38:08.312685 1996 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 19:38:08.315099 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4214caa28574754b61aad89471c77b9"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:08.316942 1996 raft_consensus.cc:595] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:08.368974 3201 raft_consensus.cc:1050] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:08.311200 3241 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
W20260927 19:38:08.426635 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 94153.7us
I20260927 19:38:08.442027 3176 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:08.444702 2141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c802eef7b2904ec984925e3999cff361"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
W20260927 19:38:08.454571 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 174072us
I20260927 19:38:08.456120 3214 raft_consensus.cc:979] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:08.448141 2141 raft_consensus.cc:595] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:08.474692 3262 raft_consensus.cc:1067] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
W20260927 19:38:08.568243 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 86029.2us
I20260927 19:38:08.577523 2067 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:08.572610 3221 raft_consensus.cc:979] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: : Instructing follower 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:08.598145 3012 raft_consensus.cc:1067] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Signalling peer 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:08.606566 2685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:08.607355 2685 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:08.607787 2685 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:08.608824 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 413361us
I20260927 19:38:08.612949 3241 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:08.619009 1994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:60296
I20260927 19:38:08.619621 1994 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:08.620012 1994 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:08.481550 3263 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.121s
I20260927 19:38:08.476924 3201 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:08.608867 2065 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:08.681036 3247 raft_consensus.cc:979] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:08.681579 2065 raft_consensus.cc:595] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:08.684504 2685 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:08.705550 1994 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:08.711948 1994 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:08.713864 2685 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:08.727346 3256 log.cc:924] Time spent T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20260927 19:38:08.681593 3247 raft_consensus.cc:1067] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
W20260927 19:38:08.746871 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 103833us
W20260927 19:38:08.759078 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 132974us
W20260927 19:38:08.761101 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45377.7us
I20260927 19:38:08.729641 3221 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:08.765782 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:08.766889 1996 raft_consensus.cc:595] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:08.774547 2685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4214caa28574754b61aad89471c77b9"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:08.775395 2685 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:08.775929 2685 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:08.789863 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:08.790860 2068 raft_consensus.cc:3032] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:08.791234 2068 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:08.798518 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:08.800452 2142 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:08.800926 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:08.806185 1992 raft_consensus.cc:3032] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:08.806836 1992 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:08.807896 1992 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:08.807682 2068 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:08.841516 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 192421us
W20260927 19:38:08.801463 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 224591us
I20260927 19:38:08.801935 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:38:08.771587 3221 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:08.846052 2068 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:08.847461 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:08.848374 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:08.849159 2685 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:08.850422 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:08.850993 2140 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:08.878023 2685 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:08.896107 3267 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20260927 19:38:08.900585 2140 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:08.915004 2142 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:08.929450 2068 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:08.939109 3221 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:08.905558 1988 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:08.950253 3276 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:38:08.974025 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 158166us
I20260927 19:38:09.058137 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:09.084019 2685 raft_consensus.cc:1226] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
W20260927 19:38:09.101275 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 227637us
I20260927 19:38:09.102645 2654 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:09.095186 1993 raft_consensus.cc:3032] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
W20260927 19:38:09.119084 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 248524us
W20260927 19:38:09.130100 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 258683us
I20260927 19:38:09.094601 3247 consensus_queue.cc:1046] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260927 19:38:09.208582 3281 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:09.116508 1993 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:09.219655 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 90951.8us
I20260927 19:38:09.238768 1962 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:09.282230 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 423648us
I20260927 19:38:09.290081 3288 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:09.290866 3288 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:09.292033 3288 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:09.400936 2068 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:09.392021 3281 raft_consensus.cc:686] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:09.050699 2684 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.418963 2687 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:09.421937 3261 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:09.424789 2143 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.427939 2141 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260927 19:38:09.476125 3098 raft_consensus.cc:979] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:09.476661 3098 raft_consensus.cc:1067] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
W20260927 19:38:09.485109 3282 log.cc:924] Time spent T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.273s user 0.003s sys 0.000s
I20260927 19:38:09.486727 2685 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.493532 2143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:38:09.494143 2143 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20260927 19:38:09.493535 3290 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260927 19:38:09.494447 2143 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:09.499213 3133 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:09.513646 3259 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:09.536379 2143 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:09.538645 2143 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:09.514317 3259 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:09.546854 3259 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:09.424827 3281 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:09.547071 1993 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:09.558085 1993 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:09.549079 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:09.555971 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:09.570309 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:09.571962 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 149) took 1379 ms. Trace:
I20260927 19:38:09.578590 3127 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.612138 1995 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.613933 2143 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.630897 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:09.631572 1988 raft_consensus.cc:3032] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:09.631963 1988 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:09.632797 1988 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:09.634924 1988 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:09.641124 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:09.641816 2068 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:09.576467 1891 rpcz_store.cc:276] 0927 19:38:08.186479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:08.186660 (+ 181us) service_pool.cc:227] Handling call
0927 19:38:09.566029 (+1379369us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:08.421658 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:08.422205 (+ 547us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:08.422233 (+ 28us) write_op.cc:435] Acquired schema lock
0927 19:38:08.422248 (+ 15us) tablet.cc:660] Decoding operations
0927 19:38:08.422557 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:08.422648 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:08.422679 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
0927 19:38:08.422870 (+ 191us) tablet.cc:699] Row locks acquired
0927 19:38:08.422889 (+ 19us) write_op.cc:260] PREPARE: finished
0927 19:38:08.423023 (+ 134us) write_op.cc:270] Start()
0927 19:38:08.423120 (+ 97us) write_op.cc:276] Timestamp: P: 1790537888422986 usec, L: 0
0927 19:38:08.423140 (+ 20us) op_driver.cc:331] REPLICATION: starting
0927 19:38:08.423853 (+ 713us) log.cc:842] Serialized 599 byte log entry
0927 19:38:09.436887 (+1013034us) op_driver.cc:444] REPLICATION: finished
0927 19:38:09.547676 (+110789us) write_op.cc:301] APPLY: starting
0927 19:38:09.547746 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:09.548679 (+ 933us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:09.548787 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:09.549007 (+ 220us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:09.549166 (+ 159us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0927 19:38:09.549212 (+ 46us) write_op.cc:312] APPLY: finished
0927 19:38:09.549408 (+ 196us) log.cc:842] Serialized 23 byte log entry
0927 19:38:09.549552 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:09.549629 (+ 77us) write_op.cc:454] Released schema lock
0927 19:38:09.549661 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:09.549912 (+ 251us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:09.549943 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":75585,"apply.run_cpu_time_us":1665,"apply.run_wall_time_us":2474,"num_ops":1,"prepare.queue_time_us":199324,"prepare.run_cpu_time_us":35873,"prepare.run_wall_time_us":151401,"raft.queue_time_us":517773,"raft.run_cpu_time_us":8220,"raft.run_wall_time_us":98393,"replication_time_us":1002591,"thread_start_us":581525,"threads_started":4,"wal-append.queue_time_us":203764}]]}
I20260927 19:38:09.645999 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 236) took 1031 ms. Trace:
I20260927 19:38:09.647816 1892 rpcz_store.cc:276] 0927 19:38:08.612213 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:08.612360 (+ 147us) service_pool.cc:227] Handling call
0927 19:38:09.643941 (+1031581us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2}
I20260927 19:38:09.651746 3274 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260927 19:38:09.654834 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:09.655500 2067 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:09.656347 1992 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:09.658201 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:09.659341 2142 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:09.664505 2068 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:09.667338 1993 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
W20260927 19:38:09.668573 3247 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260927 19:38:09.668732 2141 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:09.674360 2654 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:09.676767 3219 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:09.677301 3219 raft_consensus.cc:686] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:09.684085 2067 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:09.689949 1992 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:09.697705 3098 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.678225 3219 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:09.718230 2145 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.745872 2685 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:09.740291 1988 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:09.773470 2685 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:09.774865 3237 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:09.769677 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:09.778955 2111 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:09.780062 3304 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260927 19:38:09.783573 3259 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:09.784260 3259 raft_consensus.cc:686] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:09.785166 3259 consensus_queue.cc:237] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:10.254678 3216 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:10.265713 3216 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:10.328153 2685 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.003s sys 0.014s
W20260927 19:38:10.328533 2685 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s user 0.003s sys 0.014s
I20260927 19:38:10.338254 3216 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:10.570729 2653 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.691s (SENT)
W20260927 19:38:10.974043 3259 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.160s user 0.009s sys 0.042s
W20260927 19:38:10.974737 3259 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.161s user 0.009s sys 0.043s
W20260927 19:38:11.163151 2652 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.015s sys 0.055s
I20260927 19:38:11.163506 3314 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:11.163872 2652 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.015s sys 0.056s
I20260927 19:38:11.169634 3317 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.164448 3314 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:11.174713 2653 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.006s sys 0.013s
W20260927 19:38:11.216480 2653 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.645s user 0.007s sys 0.014s
W20260927 19:38:11.216696 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 645953us
I20260927 19:38:11.213125 3318 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.215487 3316 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.221769 3314 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.221865 3316 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.222791 3314 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.223455 3314 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.224516 3316 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.221086 3318 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.231783 3320 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.230997 3318 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.231873 3314 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.235766 3318 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.235281 3319 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.236488 3318 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.237417 3319 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.235137 3321 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.241882 3321 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.233712 3320 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.245230 3320 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.251745 3314 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.252302 3321 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.225116 3316 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.253541 3320 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.253796 3316 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.254132 3320 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.170478 3317 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.256443 3320 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.257081 3320 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.257624 3320 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.257927 3317 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.259685 3316 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.260021 3320 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.252511 3314 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.262735 1893 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:11.238611 3318 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.263703 3316 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.251778 3319 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.263686 3314 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.265316 3316 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.265208 1893 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:11.267629 3316 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.267305 1893 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 2 to 4, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:11.269265 3316 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.269739 3314 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.269869 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1043) took 1482 ms. Trace:
I20260927 19:38:11.269840 3316 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.270447 3314 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.270612 1988 rpcz_store.cc:276] 0927 19:38:09.787472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:09.787944 (+ 472us) service_pool.cc:227] Handling call
0927 19:38:09.788721 (+ 777us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:11.269446 (+1480725us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:11.269483 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:11.269532 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:11.269565 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:11.269644 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:11.269783 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:11.272634 3314 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.273213 3314 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.273782 3314 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.277490 3314 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.274904 3270 raft_consensus.cc:979] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: : Instructing follower 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:11.274248 3316 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.289770 3316 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.291064 3316 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.295516 3316 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.311990 2684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 200) took 1653 ms. Trace:
I20260927 19:38:11.312512 2684 rpcz_store.cc:276] 0927 19:38:09.658863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:09.665637 (+ 6774us) service_pool.cc:227] Handling call
0927 19:38:11.310948 (+1645311us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:11.311257 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:11.311521 (+ 264us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 19:38:11.311552 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:11.311601 (+ 49us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:11.311640 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:11.311704 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:11.311872 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36470656}
I20260927 19:38:11.352294 3324 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.352885 3324 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.355392 3324 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.355934 3098 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.356513 3098 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.355971 3324 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.358609 3098 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.358143 3324 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.361896 3324 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.364776 3210 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.365504 3210 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.369246 3210 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.369511 3330 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:11.369705 3324 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:11.370308 3324 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.370222 3330 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.372581 3324 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.372651 3098 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:11.372809 3330 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.373334 3098 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.375373 3098 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.375976 3098 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.376446 3098 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.378474 3098 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.378996 3098 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.379298 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.379709 3098 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.380374 2684 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:11.381884 3098 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.382308 3295 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:11.382879 3295 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.384478 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.385913 3295 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.387153 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.388881 3216 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.389496 3237 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.389453 3216 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.390125 3237 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.391940 3277 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.392248 3237 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.392778 3237 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:11.392647 3277 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.393188 3216 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.391824 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:11.389708 2110 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.134s user 0.018s sys 0.050s
W20260927 19:38:11.394222 2110 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.139s user 0.018s sys 0.051s
I20260927 19:38:11.395236 3277 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.395911 3277 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:11.396677 3212 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.397341 3132 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.397710 3231 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.397426 3212 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.397985 3132 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.398243 3231 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.399742 3332 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.400617 3132 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.400293 3332 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.401226 3132 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.401595 3211 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.401721 3132 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.402078 3277 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.402779 3332 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.403203 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.403648 3132 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.403882 2140 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.400326 3212 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.404917 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.405730 2137 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.406303 3313 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.406526 2145 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:11.406844 3313 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.393250 3237 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.408432 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.409211 2137 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.409255 3313 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.409886 3237 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.410149 3313 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.410430 3237 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.402100 3211 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.410805 3313 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.413113 3313 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.413324 3259 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:11.413687 3313 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.413918 3259 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.414191 3313 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.413146 3211 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.416088 3259 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.416389 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 150) took 1648 ms. Trace:
I20260927 19:38:11.400732 3231 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.393806 3216 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.417188 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.426888 3211 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.419948 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.423460 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.419560 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.441938 2142 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.441000 2140 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.441550 3216 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.442173 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.442896 2138 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.422940 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.443075 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.426121 3259 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.410957 3237 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.443886 2136 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.443953 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.444689 2684 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:11.445901 2145 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:11.445744 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.425320 3277 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.447600 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.427287 3295 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.446192 3237 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.454676 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.451587 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.452286 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.456382 2685 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:11.456560 2686 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:11.457953 1959 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:11.458645 3098 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.458994 3098 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.459995 3098 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.459203 3329 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.449465 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.462328 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.463639 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.426692 3212 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.464548 2687 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:11.416872 1893 rpcz_store.cc:276] 0927 19:38:09.768446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:09.768822 (+ 376us) service_pool.cc:227] Handling call
0927 19:38:11.416320 (+1647498us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:11.276630 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:11.276990 (+ 360us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:11.277010 (+ 20us) write_op.cc:435] Acquired schema lock
0927 19:38:11.277020 (+ 10us) tablet.cc:660] Decoding operations
0927 19:38:11.277646 (+ 626us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:11.277727 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:11.277753 (+ 26us) tablet.cc:683] Acquiring locks for 3 operations
0927 19:38:11.278063 (+ 310us) tablet.cc:699] Row locks acquired
0927 19:38:11.278080 (+ 17us) write_op.cc:260] PREPARE: finished
0927 19:38:11.278217 (+ 137us) write_op.cc:270] Start()
0927 19:38:11.278338 (+ 121us) write_op.cc:276] Timestamp: P: 1790537891278177 usec, L: 0
0927 19:38:11.278359 (+ 21us) op_driver.cc:331] REPLICATION: starting
0927 19:38:11.278935 (+ 576us) log.cc:842] Serialized 1424 byte log entry
0927 19:38:11.283089 (+ 4154us) op_driver.cc:444] REPLICATION: finished
0927 19:38:11.291101 (+ 8012us) write_op.cc:301] APPLY: starting
0927 19:38:11.291205 (+ 104us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:11.291355 (+ 150us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:11.291531 (+ 176us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:11.292040 (+ 509us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:11.292238 (+ 198us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0927 19:38:11.292288 (+ 50us) write_op.cc:312] APPLY: finished
0927 19:38:11.292536 (+ 248us) log.cc:842] Serialized 35 byte log entry
0927 19:38:11.293573 (+ 1037us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:11.293673 (+ 100us) write_op.cc:454] Released schema lock
0927 19:38:11.293718 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:11.296234 (+ 2516us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:11.296265 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":7740,"apply.run_cpu_time_us":2403,"apply.run_wall_time_us":5492,"num_ops":3,"prepare.queue_time_us":1028,"prepare.run_cpu_time_us":3434,"prepare.run_wall_time_us":3447,"raft.queue_time_us":1303,"raft.run_cpu_time_us":7033,"raft.run_wall_time_us":10427,"replication_time_us":4646,"thread_start_us":14155,"threads_started":4,"wal-append.queue_time_us":1079}]]}
I20260927 19:38:11.470429 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.472214 2654 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.425920 3132 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.479889 2654 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.480400 3314 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.466038 3212 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.457473 3259 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.416088 3313 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.485451 2108 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.479473 3210 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.487116 3098 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.439685 3211 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.488615 2654 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.479372 3320 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260927 19:38:11.492668 2141 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.451445 3295 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.482882 3216 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.482954 3314 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.485724 3237 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.500711 3323 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.482784 3277 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.485327 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.518625 3336 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.502626 3098 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.482725 3332 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.519470 3336 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.519925 3332 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.504495 3335 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.490388 3210 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.499790 3321 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.528210 3321 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.528792 3321 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.506050 3259 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.507184 3314 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.524435 3098 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.526379 3212 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.526962 3295 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.527551 3210 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.460968 3329 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.482605 3132 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.522931 3237 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.531621 3098 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.502120 3313 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.539248 3313 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.539423 3211 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.539710 3313 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:11.534227 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 48870.9us
I20260927 19:38:11.503436 3324 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.538636 3333 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.538576 3237 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.547477 2108 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:11.548965 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 61415.2us
I20260927 19:38:11.551278 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:11.551977 2687 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.552099 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.552771 2686 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:11.532867 3323 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.548508 3324 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.556313 3098 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.556739 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.556839 3098 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:11.541788 3132 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.557446 2686 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:11.557509 3098 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.559352 3294 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.559805 3294 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.560199 3294 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.531597 3277 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.563632 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.566709 2686 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:11.562034 2108 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.554961 3330 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.569430 2108 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:11.565970 2687 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:11.568919 2686 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:11.573571 3329 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.556202 3323 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.568102 3098 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.571591 3132 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.572329 3314 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.546993 3335 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.576812 3210 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.558678 3336 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.562357 3332 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.657676 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:11.589741 3132 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:11.701252 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 1.93142e+06us
I20260927 19:38:11.556589 3333 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.583165 3330 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.724519 3334 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:11.725172 3334 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.706746 3132 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.747692 2654 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:11.748826 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 167642us
I20260927 19:38:11.748620 3318 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.747519 3220 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:11.766301 3318 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.783793 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.784511 2686 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:11.786356 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.711781 3333 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.789052 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:11.776132 3318 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:11.790594 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 88105.4us
I20260927 19:38:11.794200 3098 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.794708 3098 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.802748 3132 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.805295 3337 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:11.807801 3337 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.808203 3337 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:11.808224 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:11.787309 3336 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.787122 3329 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.792318 3324 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.807230 3287 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.818254 3277 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:11.858873 3335 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.817530 3333 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.839284 3339 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.839360 3324 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.837484 3314 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.865422 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:11.830524 3329 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.876504 1993 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:11.827572 3336 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.877938 3324 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.868054 3334 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.885912 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:11.869947 3340 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:11.870571 3335 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:11.882428 3330 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.893087 3313 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.873272 3339 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.898090 3318 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:11.901813 1959 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:11.905661 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:11.906339 1993 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.906667 3324 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.872812 3333 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.892108 3343 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:38:12.048580 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 148908us
I20260927 19:38:12.012735 3343 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.101306 3343 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:11.904597 1988 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:11.892166 3321 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.097584 3324 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:11.901592 3335 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.098428 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.184666 3324 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:12.220423 3342 log.cc:924] Time spent T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.360s user 0.001s sys 0.001s
W20260927 19:38:12.237067 3323 consensus_meta.cc:287] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.621s user 0.009s sys 0.002s
I20260927 19:38:12.237705 3323 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.240324 3323 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.184988 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.246265 1987 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:12.247659 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:12.272804 3339 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.229229 3345 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:11.907449 3340 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.284206 3340 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.286105 3335 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.287281 3333 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.288720 3321 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.289923 3318 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.290956 3313 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:12.305579 3350 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260927 19:38:12.308225 3345 raft_consensus.cc:686] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:12.309703 3345 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.280124 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.312544 3329 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.301178 3337 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.361740 1988 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:12.363045 3331 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.363782 3331 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.375599 3324 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.393991 3337 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.373251 3331 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:12.500571 3324 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.501336 3324 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.504528 3324 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.505493 3324 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.506559 3324 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:12.516264 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.000s sys 0.011s
I20260927 19:38:12.516702 3329 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:12.517251 3329 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.519395 3329 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:12.516696 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.609s user 0.000s sys 0.011s
I20260927 19:38:12.520524 3329 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.521044 3329 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.523334 3329 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.524313 3345 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.524834 3345 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.526834 3345 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.527287 3345 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.527729 3345 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.528950 3324 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.530361 3345 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.534116 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.534394 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.534686 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.535097 1987 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:12.535971 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.536563 1987 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:12.536689 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.537689 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.539091 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.539722 2141 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:12.539176 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.540169 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.540863 1988 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:12.600030 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.600637 2111 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:11.806257 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.605993 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.606449 2111 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.608378 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.609738 2111 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.602041 2141 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:12.611488 3337 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:12.612231 3337 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:12.612506 3331 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:12.617511 3332 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:12.621136 3331 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.621433 3332 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.622582 3332 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:12.620477 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:48542 (request call id 838) took 2983 ms. Trace:
I20260927 19:38:12.614530 3210 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.621835 3331 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:12.627568 1962 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:12.612797 3337 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:12.632090 3345 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:12.633944 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.634361 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:12.634912 2141 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:12.635160 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.637159 3345 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:12.638760 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.639576 1962 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.649565 3345 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:12.650101 3345 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.650589 3345 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:12.624147 2070 rpcz_store.cc:276] 0927 19:38:09.636508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:09.636651 (+ 143us) service_pool.cc:227] Handling call
0927 19:38:12.609047 (+2972396us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540009f454. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79188a7c6e 00007f7911f9e807 00007f7911f9e792 00007f7911f9e6c1 00007f7911f9e63c 00007f7911f9d8f2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:12.620002 (+ 10955us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4911684480}
I20260927 19:38:12.661916 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.662865 2141 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:12.664291 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.666301 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.669947 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.671075 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.672176 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.673461 2136 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:12.675468 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.677591 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.679524 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.680467 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.682374 3337 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.689517 3337 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.691847 3098 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.669947 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:11.797008 3127 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.693822 2142 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:12.694060 3127 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.697130 3127 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.685509 3332 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:11.869513 3287 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.701184 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.701500 3332 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:12.702041 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.703634 3287 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.704078 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.706508 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:12.707253 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.708005 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.708207 1995 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:12.708891 1991 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:11.806947 3132 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:11.871827 3286 raft_consensus.cc:1067] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Signalling peer 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:12.709311 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.710323 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.715617 1988 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:12.717206 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.716178 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.718642 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.718755 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.719417 1992 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:12.719933 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.720392 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.711265 1990 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:12.710408 2111 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.727792 1992 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:12.766732 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:12.770852 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 60607us
I20260927 19:38:12.766048 3295 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 19:38:12.765094 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 971140us
I20260927 19:38:12.767251 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.728806 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.780336 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.779726 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.724167 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.762079 2137 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.721984 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.710376 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.790191 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.790556 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.791406 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.792155 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.778025 3295 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.732635 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.728757 3348 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:12.757828 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:12.728235 3346 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.733559 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46930 (request call id 73) took 1342 ms. Trace:
I20260927 19:38:12.731894 3345 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.798879 2144 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:12.797989 3295 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:12.804620 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.810084 1996 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:12.812105 2111 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.807530 1990 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:12.815083 1995 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:12.816725 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:12.792892 3215 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:12.821208 3277 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:12.821667 3277 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.822281 3277 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:12.829187 1992 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:12.827711 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.828231 3348 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.833030 2142 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:12.828790 1989 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:12.841240 3344 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.842360 3344 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:12.844376 1962 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.801481 2138 rpcz_store.cc:276] 0927 19:38:11.391429 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:11.391571 (+ 142us) service_pool.cc:227] Handling call
0927 19:38:12.733066 (+1341495us) spinlock_profiling.cc:243] Waited 48.8 ms on lock 0x7b54001c5054. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:12.733331 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112221952}
I20260927 19:38:12.846177 1962 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.853726 3336 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.854990 1991 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:12.863600 3287 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:12.867455 3287 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:12.869550 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 61042.9us
I20260927 19:38:12.869159 3270 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.870838 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.870981 3270 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.875563 3287 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.876600 3127 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.876812 3344 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.877339 3127 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.880349 2142 raft_consensus.cc:1226] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.878064 3225 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.882311 3225 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.857080 1893 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:12.883107 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.884639 3225 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.878597 3358 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:12.886329 3324 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:12.886834 3324 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.887279 3324 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:12.887629 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:12.882437 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.891037 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:12.891639 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:12.857858 3345 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:12.892546 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800148100 after lost signal to thread 821
I20260927 19:38:12.891033 1995 raft_consensus.cc:1226] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.893990 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:12.894320 2166 debug-util.cc:398] Leaking SignalData structure 0x7b08001e44e0 after lost signal to thread 1927
W20260927 19:38:12.894876 2166 debug-util.cc:398] Leaking SignalData structure 0x7b08001d96c0 after lost signal to thread 1942
W20260927 19:38:12.896733 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800299580 after lost signal to thread 2020
I20260927 19:38:12.898178 3286 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:11.791975 3220 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.898949 3286 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.899346 3220 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.900597 3211 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:12.901892 3211 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:12.902565 3211 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.881494 1989 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:60296
I20260927 19:38:12.902334 3358 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.904392 1989 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:11.806470 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:12.907678 2686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34940 (request call id 220) took 1102 ms. Trace:
I20260927 19:38:12.855343 1992 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:12.884358 1893 catalog_manager.cc:5495] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:12.908001 2686 rpcz_store.cc:276] 0927 19:38:11.805043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:11.806232 (+ 1189us) service_pool.cc:227] Handling call
0927 19:38:12.907496 (+1101264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:12.909678 3338 raft_consensus.cc:2741] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260927 19:38:12.911408 3286 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:12.912922 2166 debug-util.cc:398] Leaking SignalData structure 0x7b08002c4720 after lost signal to thread 2095
I20260927 19:38:12.912849 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:12.913478 2166 debug-util.cc:398] Leaking SignalData structure 0x7b080022bae0 after lost signal to thread 2096
I20260927 19:38:12.913951 2684 raft_consensus.cc:2379] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:12.889106 3127 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.917177 3211 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.918737 3225 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.919595 3359 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.919761 3225 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.920615 3359 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.921649 3286 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.922348 2687 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.921677 3356 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:12.924006 3356 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.924691 3220 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.923324 3359 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.926690 3356 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.926733 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.906872 1989 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:12.936179 2111 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.937924 3270 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.940552 3358 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.899704 1990 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:12.922777 3286 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.946703 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:12.942484 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800368320 after lost signal to thread 2169
I20260927 19:38:12.947651 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:12.953688 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.953672 3259 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:12.955789 3259 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.956270 3259 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:12.956401 2070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:12.956669 3356 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.827239 3346 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.960541 2070 raft_consensus.cc:595] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:12.946964 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.941320 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.960631 3356 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.964514 3012 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.966544 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.967110 3012 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.968175 3348 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.968626 3356 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:12.944435 3344 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.965449 2688 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:12.969996 3344 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.972663 3344 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.974028 2683 raft_consensus.cc:1226] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.860306 3331 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:12.977959 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 90880.6us
I20260927 19:38:12.980227 3287 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.981091 2688 raft_consensus.cc:1226] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.975111 3012 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:12.950659 3359 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.977936 2111 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:12.980942 3287 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:12.987211 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:12.988082 3211 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:12.988515 3211 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:12.986003 3224 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.989045 3211 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:12.978130 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:12.959097 3220 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:12.988619 2680 raft_consensus.cc:1226] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.998147 2144 raft_consensus.cc:2379] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:12.998471 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:12.992087 3344 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:12.993251 2063 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:12.999334 2066 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:12.994892 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:12.999884 3344 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.000901 2139 raft_consensus.cc:1226] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:12.999369 3324 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.001528 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.000924 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.003077 3324 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.003523 2068 raft_consensus.cc:2362] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
W20260927 19:38:13.003753 3294 consensus_meta.cc:287] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.443s user 0.012s sys 0.026s
I20260927 19:38:13.010953 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.013653 3360 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:13.014447 3360 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.014802 3360 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.013594 1962 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.019342 3338 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.019793 3338 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.020217 3338 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.020176 1962 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.021524 3330 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:13.022111 3330 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.022729 3330 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.030376 3345 consensus_queue.cc:1046] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:13.031559 3098 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.031899 3098 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:13.032799 3098 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.033529 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:13.034345 3098 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.035000 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.035781 2144 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.039140 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:13.005587 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800369120 after lost signal to thread 2712
I20260927 19:38:13.008723 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:13.006089 3331 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.044417 2144 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:13.049604 3344 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.043447 3012 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:13.027581 3334 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:12.989429 3224 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.004185 3294 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:12.996995 3359 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.068082 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.057860 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.067718 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:13.082417 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 74053us
I20260927 19:38:13.068528 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.008529 3287 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.095153 3012 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.078419 1959 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:13.068912 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.069396 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.066861 1962 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:13.078763 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800369020 after lost signal to thread 3171
I20260927 19:38:13.068177 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.097400 2063 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:13.072893 3356 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:13.101531 3098 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:13.103061 2068 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:13.073155 3348 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:13.103909 3012 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.097190 3210 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.079993 3286 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.045611 3220 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.107687 3098 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:13.008514 3316 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260927 19:38:13.108228 3012 consensus_queue.cc:1046] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260927 19:38:13.102056 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800369000 after lost signal to thread 3175
I20260927 19:38:13.067708 3231 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:38:13.114684 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 62929.8us
I20260927 19:38:13.114638 2064 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:13.116333 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.074522 3225 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.120998 1959 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:13.069898 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.107450 3210 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.121989 2654 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:13.067224 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.125795 3346 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.110069 3358 consensus_queue.cc:1046] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260927 19:38:13.126272 2111 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:13.114755 2166 debug-util.cc:398] Leaking SignalData structure 0x7b0800368fe0 after lost signal to thread 3176
I20260927 19:38:13.121140 3287 raft_consensus.cc:3032] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.123736 3210 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.123831 3336 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.126956 3231 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.129926 3294 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.124462 3348 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.142311 3259 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.143867 3348 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.133702 3358 raft_consensus.cc:3032] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.064844 3360 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.136070 3336 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.136897 3334 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:13.137089 1959 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:13.137909 3220 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.137766 2062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:13.153440 3331 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:13.153554 2062 raft_consensus.cc:595] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:13.154045 3277 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.154713 1959 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:13.155113 3329 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:13.137810 3098 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.140672 3316 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.160878 3316 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.161387 3316 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.161576 2142 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63522eab31e04c8191338bb22acb0297"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:13.162242 2142 raft_consensus.cc:595] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:38:13.141441 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 90948.4us
I20260927 19:38:13.107321 3211 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.154085 3331 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.157963 2137 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.149902 3358 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.150218 3336 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.158651 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.134100 3359 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.169219 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.108946 3338 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.134402 3287 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.115840 3356 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.168979 3331 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.178359 3329 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:13.166426 3224 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.192031 2067 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:13.193909 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.149946 3330 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.194567 2070 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:13.184907 2062 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:13.205920 3358 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.207674 3358 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.188259 3231 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.183195 3334 consensus_queue.cc:1046] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260927 19:38:13.192166 3359 consensus_queue.cc:1046] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260927 19:38:13.212014 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.214995 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.195710 2065 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:13.212456 3359 raft_consensus.cc:3032] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.225538 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.215016 3215 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40144 (request call id 633) took 3552 ms. Trace:
W20260927 19:38:13.226711 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 124423us
I20260927 19:38:13.228197 3132 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:13.225042 3277 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.235339 3132 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.236297 2682 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:13.219475 3211 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.222321 2065 raft_consensus.cc:595] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:13.244678 3338 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.245443 3360 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.241879 3287 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.242115 3132 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.245918 3316 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.246915 3287 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.247404 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.223649 3359 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.243140 2111 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.252254 3316 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.258447 3132 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.259766 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.264850 1988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4214caa28574754b61aad89471c77b9"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:13.265710 1988 raft_consensus.cc:595] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:13.266157 1988 raft_consensus.cc:603] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:13.258935 3132 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.267935 3132 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.270081 1988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:13.271008 1988 raft_consensus.cc:595] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:13.271727 3329 raft_consensus.cc:3032] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.271253 3359 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.272152 3329 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:13.272682 3359 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.272989 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:38:13.275432 1989 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:13.277503 2653 leader_election.cc:334] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.691s (SENT)
I20260927 19:38:13.277549 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:13.263703 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:13.309144 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 142774us
I20260927 19:38:13.315693 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:13.253723 3210 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.236377 3215 rpcz_store.cc:276] 0927 19:38:09.658808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:09.659209 (+ 401us) service_pool.cc:227] Handling call
0927 19:38:09.671439 (+ 12230us) log.cc:842] Serialized 501 byte log entry
0927 19:38:11.409701 (+1738262us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b2c00063708. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912354ada 00007f7912354763 00007f7912354701 00007f791399ed62 00007f791399ef05 00007f79138f3f91 00007f79138dc545 00007f79138e1771 00007f79138e16f9 00007f79138e1689
0927 19:38:11.411207 (+ 1506us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b180020e980. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f791399ef11 00007f79138f3f91 00007f79138dc545 00007f79138e1771 00007f79138e16f9 00007f79138e1689 00007f79138e1651 00007f79138e081d 00007f7919c9e204 00007f7919c9e039
0927 19:38:13.201687 (+1790480us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35142,"raft.queue_time_us":28245,"raft.run_cpu_time_us":29878,"raft.run_wall_time_us":1689320,"spinlock_wait_cycles":14581248,"thread_start_us":50698,"threads_started":1,"wal-append.queue_time_us":88576,"wal-append.run_cpu_time_us":1495,"wal-append.run_wall_time_us":1001492}
W20260927 19:38:13.322641 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 74287.7us
I20260927 19:38:13.273137 3329 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.327087 3329 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.334221 2680 raft_consensus.cc:1226] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:13.334822 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:13.339089 3295 consensus_meta.cc:287] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.528s user 0.010s sys 0.002s
I20260927 19:38:13.339522 3295 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.340339 2682 raft_consensus.cc:1226] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:13.341552 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f19d4799cf134d51b543a2952f564adc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:13.318898 3356 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.342567 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:13.342587 2145 raft_consensus.cc:2379] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.318589 3316 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.318485 3231 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:13.351096 2111 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.348276 3337 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:13.321950 3330 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.355613 3231 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.358577 3277 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:13.359109 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:13.360159 2654 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:13.361119 1989 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.362912 2111 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.377027 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:13.391748 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.393008 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.395133 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.401140 3337 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:13.401890 3337 raft_consensus.cc:686] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:13.394477 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.401578 2067 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.392573 2070 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:13.401969 2064 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.403156 3337 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.404496 3277 raft_consensus.cc:686] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:13.405522 3277 consensus_queue.cc:237] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.396392 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.407248 2062 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.402902 3364 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:13.409229 3364 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.409672 3364 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.402907 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.374711 3132 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.411314 2063 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.349320 3316 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:13.412371 2654 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.415630 3316 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.417256 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:13.417830 3366 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:13.418301 3366 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:13.418723 3366 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.419142 1959 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:13.353552 3295 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.379861 3231 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.378289 2680 raft_consensus.cc:1226] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
W20260927 19:38:13.436672 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 320345us
I20260927 19:38:13.378568 3259 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.390244 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.447763 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.449235 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.448546 1988 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.370539 3336 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.452260 1959 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:13.453263 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.454197 1993 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:13.455111 3338 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:13.455354 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:13.455869 3098 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:13.448686 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.457648 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.464283 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:13.458560 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.459194 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:13.376518 3331 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.453142 3336 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.438405 3210 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.463027 3334 raft_consensus.cc:979] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:13.438026 3330 raft_consensus.cc:2741] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260927 19:38:13.477914 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.482702 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.484305 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:13.489045 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 72292.8us
I20260927 19:38:13.492834 2143 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.491737 2063 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.495841 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.496819 2142 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.498896 2139 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.480576 1987 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.481429 3359 consensus_queue.cc:1046] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260927 19:38:13.480625 2069 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.498845 3358 consensus_queue.cc:1046] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260927 19:38:13.514092 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.514735 2143 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.517421 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.517701 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.518404 2141 raft_consensus.cc:2379] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.467936 3316 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.507570 3334 raft_consensus.cc:1067] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:13.528685 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.531726 2142 raft_consensus.cc:2379] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.535575 2145 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:13.538439 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.540678 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.542747 3132 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.545463 2143 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:13.505718 3210 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260927 19:38:13.552554 2687 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:13.560741 3331 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.570082 2064 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:13.572909 2066 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:13.579017 2687 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:13.579475 2687 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:13.579692 2687 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.553004 3359 consensus_queue.cc:1046] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
W20260927 19:38:13.580389 3369 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.118s user 0.002s sys 0.000s
I20260927 19:38:13.574503 3358 consensus_queue.cc:1046] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260927 19:38:13.568742 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.595666 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.576759 1988 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.597263 3270 raft_consensus.cc:3032] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.614629 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.615139 3316 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.616222 3287 raft_consensus.cc:3032] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.616649 3287 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.616951 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:13.617746 1991 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.617901 3356 raft_consensus.cc:3032] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.618263 3356 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.614687 3270 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.619141 3356 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.619570 3270 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.620646 3356 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.620883 3270 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.626669 2111 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.628758 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:13.629793 3224 raft_consensus.cc:3032] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.629519 3287 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.630152 3224 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.629966 3211 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:13.641994 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.645448 3215 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:13.645891 3224 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.645951 3215 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:13.647568 3224 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.647998 3215 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.635555 2687 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.632465 2068 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:13.657989 2680 raft_consensus.cc:1226] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:13.661350 3368 consensus_queue.cc:1046] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:13.670367 2687 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.679893 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.634536 3287 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.681291 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.682202 2066 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.662528 3345 raft_consensus.cc:3032] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.689779 2139 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:13.663059 3211 raft_consensus.cc:686] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:13.690907 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.691120 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.692808 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.692955 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.693174 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.783551 2064 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.782438 2063 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:13.671376 3366 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.795195 3211 consensus_queue.cc:237] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:13.691769 3345 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:13.798964 1987 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:13.799011 3360 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:13.799444 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.799813 3360 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.718603 3368 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260927 19:38:13.804118 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:13.805375 3360 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.808176 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:13.809106 2139 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:13.810462 2683 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:13.810822 1962 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:13.812212 3366 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:13.818841 3220 consensus_queue.cc:1046] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20260927 19:38:13.820535 2110 leader_election.cc:334] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.859s (SENT)
I20260927 19:38:13.821334 3220 raft_consensus.cc:3032] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:13.821755 3220 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.812119 3360 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:13.822589 3220 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.823503 3360 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:13.823900 3360 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.824066 3220 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.799103 3345 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.768396 3364 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.837522 2685 raft_consensus.cc:2379] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:13.836483 3345 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.838822 3364 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:13.838814 2683 raft_consensus.cc:1226] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:13.840939 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.842056 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.842098 2062 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.842319 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.842494 3359 consensus_queue.cc:1046] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:13.842739 2064 raft_consensus.cc:2362] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2. Re-sending same reply.
I20260927 19:38:13.843050 2069 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.843576 3359 raft_consensus.cc:3032] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:13.843645 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.844022 3359 raft_consensus.cc:729] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:13.721624 3215 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.844442 2068 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.692863 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.848414 2065 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.844899 3359 consensus_queue.cc:260] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.844116 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.849812 3359 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:13.850184 2069 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:13.846860 3215 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.856721 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.862818 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.865034 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.865932 2063 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.868109 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.868956 2070 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.872924 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.889676 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.890460 2062 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.891139 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.897820 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.898176 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.897801 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.900580 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:13.861642 3360 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:13.911363 1990 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:13.911482 2063 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.905786 2071 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.909301 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.912931 2068 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:13.914913 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.915022 3360 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:13.915985 2070 raft_consensus.cc:2379] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:13.917728 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.919214 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:13.919355 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:13.920034 2139 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:13.924094 2110 leader_election.cc:334] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.013s (SENT)
I20260927 19:38:13.925345 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.926215 2065 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.924533 2110 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:13.931181 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.931488 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.939245 2139 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:13.940929 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.940989 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.984834 2071 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.940225 1989 raft_consensus.cc:1261] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:13.986299 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.986382 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.956323 3211 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:13.987380 2062 raft_consensus.cc:2379] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:13.988199 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.991846 2070 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.993214 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.990139 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.994261 2062 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.996829 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.949800 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.999414 2068 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:13.997700 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:13.956194 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47876us
I20260927 19:38:13.997886 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.001518 1962 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:14.002070 2062 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:14.001766 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:13.998840 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:13.999331 3294 consensus_queue.cc:1035] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:13.970959 1990 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:14.005295 2070 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:14.007339 3360 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:14.006527 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.018213 2069 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:14.019040 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 495644us
I20260927 19:38:14.021742 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:14.024298 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:14.027117 3332 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:14.027721 3332 raft_consensus.cc:686] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:14.027743 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.016922 3360 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:14.030548 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.032855 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.033594 2070 raft_consensus.cc:2362] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2. Re-sending same reply.
I20260927 19:38:14.034401 3332 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.028584 2062 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:14.030119 3390 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:14.040788 3390 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:14.042512 3360 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.044831 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.056600 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:14.042876 3390 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:14.089468 3364 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:14.089593 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 252) took 1509 ms. Trace:
I20260927 19:38:14.093523 2654 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:14.095046 1990 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:14.096691 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.098234 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.098846 2068 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:14.101806 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.101505 1892 catalog_manager.cc:5495] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:14.102983 2070 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:14.103284 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.104031 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.104980 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.102998 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.106081 2071 raft_consensus.cc:3032] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:14.106611 2071 raft_consensus.cc:729] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:14.107470 2071 consensus_queue.cc:260] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:14.108397 2071 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:14.106124 2066 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.115307 1991 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.094936 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.101221 2064 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.090227 1893 rpcz_store.cc:276] 0927 19:38:12.581053 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:12.583257 (+ 2204us) service_pool.cc:227] Handling call
0927 19:38:14.089449 (+1506192us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:13.120969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:13.125782 (+ 4813us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:13.125864 (+ 82us) write_op.cc:435] Acquired schema lock
0927 19:38:13.125902 (+ 38us) tablet.cc:660] Decoding operations
0927 19:38:13.126654 (+ 752us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:13.126814 (+ 160us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:13.126878 (+ 64us) tablet.cc:683] Acquiring locks for 2 operations
0927 19:38:13.128906 (+ 2028us) tablet.cc:699] Row locks acquired
0927 19:38:13.128945 (+ 39us) write_op.cc:260] PREPARE: finished
0927 19:38:13.129560 (+ 615us) write_op.cc:270] Start()
0927 19:38:13.129735 (+ 175us) write_op.cc:276] Timestamp: P: 1790537893129485 usec, L: 0
0927 19:38:13.129771 (+ 36us) op_driver.cc:331] REPLICATION: starting
0927 19:38:13.187484 (+ 57713us) log.cc:842] Serialized 987 byte log entry
0927 19:38:13.938185 (+750701us) op_driver.cc:444] REPLICATION: finished
0927 19:38:14.034876 (+ 96691us) write_op.cc:301] APPLY: starting
0927 19:38:14.034984 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:14.035111 (+ 127us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:14.035252 (+ 141us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:14.042153 (+ 6901us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:14.042401 (+ 248us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0927 19:38:14.042466 (+ 65us) write_op.cc:312] APPLY: finished
0927 19:38:14.042763 (+ 297us) log.cc:842] Serialized 29 byte log entry
0927 19:38:14.042949 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:14.043042 (+ 93us) write_op.cc:454] Released schema lock
0927 19:38:14.043103 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:14.043386 (+ 283us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:14.043413 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":96303,"apply.run_cpu_time_us":2707,"apply.run_wall_time_us":8807,"num_ops":2,"prepare.queue_time_us":116140,"prepare.run_cpu_time_us":41400,"prepare.run_wall_time_us":193172,"raft.queue_time_us":211956,"raft.run_cpu_time_us":21646,"raft.run_wall_time_us":86891,"replication_time_us":808319,"thread_start_us":415312,"threads_started":4,"wal-append.queue_time_us":157870}]]}
W20260927 19:38:14.144586 1880 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3d1bc2cd9efe43399ddba396f5ddff85 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.148804 2071 raft_consensus.cc:1261] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:14.142339 3390 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.133926 3372 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:14.172189 3372 raft_consensus.cc:686] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:14.173776 3390 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:14.173638 3372 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.181638 2108 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:14.182765 3211 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:14.187511 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:14.191169 1996 raft_consensus.cc:2379] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:14.193094 2139 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.199868 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:14.201279 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:14.201530 1988 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.206393 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:14.204947 3332 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:14.215049 3332 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.215692 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:14.212687 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 165925us
I20260927 19:38:14.217878 2139 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:14.212800 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:14.213591 1996 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.224520 3332 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:14.224602 2137 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:14.231282 3334 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:14.231580 1991 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:14.238830 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.248646 2682 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260927 19:38:14.250481 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:14.255820 1993 raft_consensus.cc:2379] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:14.251374 1992 raft_consensus.cc:1261] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:14.251860 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:14.260046 1987 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:14.259768 3334 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:14.269256 1995 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:14.280859 2143 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.285496 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.325700 2071 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.280543 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:14.306002 1893 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:14.272631 2139 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:14.279701 2140 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.280300 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:14.301605 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.317790 2144 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.329130 1893 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 1 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.345232 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.351325 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:14.303599 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.278872 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:14.287319 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:14.321966 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:14.321012 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:14.359555 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:14.320147 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.368182 2069 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.322422 3211 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20260927 19:38:14.370641 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.375326 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.341084 3295 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260927 19:38:14.376011 2063 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:14.378150 2687 raft_consensus.cc:3032] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:14.378659 2687 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:14.365793 2681 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.377794 2136 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:14.379559 2687 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.360345 1987 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:14.383054 2687 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:14.385041 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.385877 2064 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.370971 2062 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.382359 2683 raft_consensus.cc:2379] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:14.393240 3360 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:14.393499 2111 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:14.393870 3360 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.394248 3405 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:14.394690 3405 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:14.395046 3405 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:14.396917 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:14.399513 2684 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:14.400419 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.400759 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.401687 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.403136 2064 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:14.406611 2687 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:14.407233 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.407536 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.315919 3334 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.421191 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.421747 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.422962 2063 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.430675 3360 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:14.432950 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.433162 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.433794 2066 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:14.433808 2069 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.443022 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.421046 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.522173 3375 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:14.443674 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.600005 3372 consensus_queue.cc:1046] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260927 19:38:14.603474 3375 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.604321 3364 raft_consensus.cc:3032] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:14.604698 3364 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:14.583447 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:14.592435 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet f19d4799cf134d51b543a2952f564adc with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60280
I20260927 19:38:14.605821 3375 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:14.605599 3364 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.606109 2687 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:14.531853 2108 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:14.637735 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 108314us
I20260927 19:38:14.638800 3364 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:14.638814 3409 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:14.639914 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.641187 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.641345 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.641893 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.641114 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.444020 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.643193 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.644248 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.602515 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.645388 2145 raft_consensus.cc:1226] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.645536 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.646553 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.604786 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.455756 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.584214 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:14.527361 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 84601.3us
I20260927 19:38:14.648396 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.641067 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.608006 2064 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:14.649375 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.450431 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.649880 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.650445 2064 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:14.584776 1891 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:14.651197 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.654268 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:14.654621 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.649659 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.655230 2064 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.655025 1891 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 3, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:14.655062 2687 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.655977 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.656646 2064 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.658584 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.660420 2144 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.660545 2143 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:14.660677 1891 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 3, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:14.666682 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:14.666589 2140 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.446281 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.670120 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.689309 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.672521 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:14.662894 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 148978us
I20260927 19:38:14.689309 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:14.703578 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.669009 1891 catalog_manager.cc:5495] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:14.690789 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.712883 2067 raft_consensus.cc:3032] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:14.713093 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.713404 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.713762 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.715520 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.713308 2067 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:14.716951 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:14.716991 2068 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:14.717783 2144 raft_consensus.cc:2379] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:14.596885 3405 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.731937 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.704489 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:14.749670 2067 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:14.760092 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 122896us
I20260927 19:38:14.685791 3324 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260927 19:38:14.766753 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:14.767290 2067 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:14.763064 2064 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.765185 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.768533 2144 raft_consensus.cc:1226] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.769860 2066 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.772661 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.779215 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.738090 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.786507 2144 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.789000 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.793398 3413 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:14.799777 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:14.786383 2137 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:14.800168 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.803855 2138 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.804293 2144 raft_consensus.cc:1226] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.810484 2067 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:14.813230 1993 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.838419 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.839723 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.813840 1991 raft_consensus.cc:1226] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.813230 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.814908 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.843568 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.844672 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.845059 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.815634 1994 raft_consensus.cc:595] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:14.816700 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0fcc7a171b3e443d8f28fd70f8742470 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:38:14.817070 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.849187 1989 raft_consensus.cc:1226] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
W20260927 19:38:14.850010 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 66482.1us
I20260927 19:38:14.850560 1989 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.851925 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:14.852146 1989 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.815737 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.813401 1995 raft_consensus.cc:1226] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.855078 2142 raft_consensus.cc:1226] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.822031 3405 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:14.800230 3411 tablet_replica.cc:330] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
W20260927 19:38:14.856074 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 73910.8us
I20260927 19:38:14.831506 1893 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 3, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:14.856407 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.857559 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.841713 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.845249 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:14.855523 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 71644.2us
I20260927 19:38:14.534584 2682 raft_consensus.cc:2932] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.856782 2139 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.857862 2136 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:14.843499 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.864482 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.862457 1893 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 3 to 4, VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130) evicted. New cstate: current_term: 1 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:14.868846 1893 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 3, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:14.869850 3413 consensus_queue.cc:1225] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer 3d1bc2cd9efe43399ddba396f5ddff85 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 19:38:14.866488 3294 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260927 19:38:14.877970 3411 raft_consensus.cc:2229] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:14.878950 3411 raft_consensus.cc:2258] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:14.853013 1990 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:14.883492 3295 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:14.884943 2108 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:14.885910 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.886701 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.887648 2110 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:14.889505 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.889979 2138 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.890085 3364 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:14.890582 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.891081 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.891288 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.891912 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.892565 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.892748 3368 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:14.897884 1993 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:14.901539 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.889111 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.871644 1893 catalog_manager.cc:5495] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:14.907944 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:14.921175 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.859982 3416 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 19:38:14.911324 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 59469us
I20260927 19:38:14.911324 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.933665 2110 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:14.935322 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:14.936323 2653 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:14.937799 2687 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:14.938171 2687 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:14.938566 1991 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:14.938856 2687 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:14.942716 2687 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:14.944003 2110 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:14.945053 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:14.947512 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:14.949221 2654 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:14.951589 2687 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:14.925405 3215 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:14.925225 3405 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:14.989050 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 44065.1us
I20260927 19:38:14.989573 3405 raft_consensus.cc:2741] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260927 19:38:14.990010 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:14.927793 3421 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:14.929765 3294 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20260927 19:38:14.994362 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 64872.2us
W20260927 19:38:14.996418 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 47261.3us
I20260927 19:38:14.998694 2653 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.000166 2654 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:14.954071 3332 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:14.988199 3215 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:14.980780 3408 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:14.986945 3324 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:14.940202 3409 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260927 19:38:14.987532 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:14.993927 3416 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:15.022135 3419 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:15.011161 3215 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:15.000823 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:15.012993 3421 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:15.013566 3324 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:38:15.045845 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 186115us
I20260927 19:38:15.020903 3408 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:15.029654 3364 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:15.042618 2138 raft_consensus.cc:595] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:15.042657 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:15.054844 3412 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260927 19:38:15.061589 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 104745us
I20260927 19:38:15.042855 3324 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:15.077296 2037 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:15.078613 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 88671.5us
I20260927 19:38:15.079794 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:15.079970 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:15.074874 3416 raft_consensus.cc:686] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:15.049738 3408 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:15.102478 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 63488.1us
I20260927 19:38:15.115482 3416 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.073947 3375 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:15.041092 3414 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
W20260927 19:38:15.127771 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47917.7us
I20260927 19:38:15.081486 3411 ts_tablet_manager.cc:1905] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:15.140625 2110 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:15.102159 3408 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:15.084296 3431 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.148957 2687 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:15.143230 3428 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.147362 3332 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260927 19:38:15.142551 3420 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:15.175381 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:15.175927 3419 raft_consensus.cc:686] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:15.176694 2687 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:15.177188 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.181547 3390 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:15.182041 3390 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:15.182936 3390 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.132216 3375 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.186524 3375 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:15.190682 2110 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:15.194591 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 114873us
W20260927 19:38:15.194943 2653 leader_election.cc:334] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.571s (SENT)
I20260927 19:38:15.195365 2653 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:15.197101 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:15.197945 2687 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
W20260927 19:38:15.242342 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47420.7us
I20260927 19:38:15.177642 3429 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 19:38:15.248446 2653 leader_election.cc:334] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.627s (SENT)
I20260927 19:38:15.248914 2653 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:15.253484 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 107838us
I20260927 19:38:15.246753 1959 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:15.258270 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 53062.3us
W20260927 19:38:15.261492 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 60063.9us
W20260927 19:38:15.266866 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 76298.7us
W20260927 19:38:15.347464 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 93319.3us
I20260927 19:38:15.205890 3405 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 19:38:15.355957 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 101806us
W20260927 19:38:15.358865 2037 leader_election.cc:341] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Illegal state: must be running to vote when last-logged opid is not known
I20260927 19:38:15.359485 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:15.244817 3364 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.361589 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:15.356370 3359 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.258599 3429 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.266403 3427 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 19:38:15.368698 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 106530us
I20260927 19:38:15.374128 2067 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:38:15.374782 2067 raft_consensus.cc:595] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:15.375041 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 126611us
I20260927 19:38:15.177068 3419 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:15.361435 3404 log.cc:924] Time spent T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
W20260927 19:38:15.384096 2653 leader_election.cc:334] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.637s (SENT)
I20260927 19:38:15.384765 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.358369 3405 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.260511 3420 consensus_queue.cc:1046] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20260927 19:38:15.356949 3421 raft_consensus.cc:686] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:15.276607 3431 consensus_queue.cc:1046] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.116s
W20260927 19:38:15.393378 2653 leader_election.cc:334] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.680s (SENT)
W20260927 19:38:15.406373 2653 leader_election.cc:334] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.726s (SENT)
I20260927 19:38:15.406839 2653 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:15.393903 3364 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:15.372795 3359 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.372488 3429 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:15.394565 3294 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:15.394713 1993 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:15.401619 3448 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.371322 3427 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:15.365939 3447 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.403157 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.432142 2110 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:15.399461 3405 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:15.431061 3437 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.475658 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:15.461179 3427 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:15.480885 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 74546.4us
I20260927 19:38:15.379412 3432 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260927 19:38:15.490200 3428 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:15.492007 2110 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:15.495584 2653 leader_election.cc:334] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.773s (SENT)
W20260927 19:38:15.511888 2653 leader_election.cc:334] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.826s (SENT)
I20260927 19:38:15.477414 3332 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:15.518148 2110 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:15.490687 3428 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:15.557512 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 45677.6us
I20260927 19:38:15.514094 3432 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:15.566004 2653 leader_election.cc:334] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.856s (SENT)
I20260927 19:38:15.568764 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:15.564165 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46147.8us
W20260927 19:38:15.529033 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 184380us
W20260927 19:38:15.587723 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 184626us
I20260927 19:38:15.588634 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:15.588917 2653 leader_election.cc:334] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.916s (SENT)
I20260927 19:38:15.528841 3406 raft_consensus.cc:3032] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
W20260927 19:38:15.595052 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 222557us
I20260927 19:38:15.529671 3428 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:15.522428 3332 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.555866 3431 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:15.526471 3425 raft_consensus.cc:2741] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20260927 19:38:15.617569 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47862.9us
I20260927 19:38:15.618589 2110 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:15.621682 1880 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3d1bc2cd9efe43399ddba396f5ddff85 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:15.528620 3215 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:15.626092 2111 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:15.591228 3390 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:15.609405 3324 consensus_meta.cc:287] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.508s user 0.007s sys 0.000s
I20260927 19:38:15.601472 3406 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:15.614786 3369 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
W20260927 19:38:15.633793 2653 leader_election.cc:334] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.932s (SENT)
I20260927 19:38:15.596915 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:15.649945 2142 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:15.641024 2110 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:15.612797 3332 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:15.647147 2145 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:15.617008 3463 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.662364 2110 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:15.679674 3451 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260927 19:38:15.646908 3324 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.682574 3324 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:15.682117 3406 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:15.690158 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 52388.4us
I20260927 19:38:15.660140 2136 raft_consensus.cc:595] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:15.690567 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 62470.5us
I20260927 19:38:15.625494 3431 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.524199 3421 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.676106 3463 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:15.715399 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:15.718618 2140 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:15.692492 3211 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.707859 3432 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:15.724038 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:15.733319 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260927 19:38:15.737488 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:15.738183 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:15.715405 3359 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:15.747823 3359 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.748610 3359 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:15.751935 3450 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.197s user 0.001s sys 0.001s
I20260927 19:38:15.752223 3464 raft_consensus.cc:3032] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:15.752867 3464 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:15.755934 3466 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:15.757694 3408 consensus_meta.cc:287] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.600s user 0.008s sys 0.003s
I20260927 19:38:15.731191 3463 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.762447 3454 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 19:38:15.769901 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 166673us
I20260927 19:38:15.779701 2037 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.750974 3432 raft_consensus.cc:686] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:15.760936 2653 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.729s (SENT) [suppressed 85 similar messages]
I20260927 19:38:15.788251 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:15.789243 2653 leader_election.cc:334] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.937s (SENT)
I20260927 19:38:15.789724 2653 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:15.790855 2653 leader_election.cc:334] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.948s (SENT)
I20260927 19:38:15.763101 3454 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.797080 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:15.798961 2653 leader_election.cc:334] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.020s (SENT)
I20260927 19:38:15.768707 3463 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:15.827023 3429 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:15.816812 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:15.831996 2653 leader_election.cc:334] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.114s (SENT)
I20260927 19:38:15.827824 3368 raft_consensus.cc:979] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:15.807753 3215 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:15.801208 3454 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:15.836366 3465 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.763809 3408 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:15.835784 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.839462 3465 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.846112 3429 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:15.846575 2145 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:15.846722 3459 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:15.837291 3463 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.882405 3463 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.883093 3463 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:15.883596 3463 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.847855 3431 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:15.852072 3467 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260927 19:38:15.756554 3464 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:15.789762 3405 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.873274 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:15.967074 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 94109.7us
I20260927 19:38:15.968082 2137 raft_consensus.cc:1226] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
W20260927 19:38:15.875711 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 43768.3us
I20260927 19:38:15.877584 3211 raft_consensus.cc:979] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:15.885766 3459 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:15.845247 1990 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:15.893741 3432 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:15.969326 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.971027 3405 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:15.971284 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:15.969882 3459 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:15.938468 2140 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:15.970789 2653 leader_election.cc:334] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.983s (SENT)
I20260927 19:38:15.956475 3406 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:15.974390 2653 leader_election.cc:334] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.212s (SENT)
I20260927 19:38:15.934921 3428 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:15.955144 2138 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:15.969548 3431 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:15.975778 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.005494 3431 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.006883 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.007580 3431 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.008752 3431 consensus_queue.cc:1046] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:15.976411 2653 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:15.996297 3465 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.009887 3431 raft_consensus.cc:3032] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:16.010941 3431 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:16.011791 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.011665 3431 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.002584 3359 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.003929 2139 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:16.012810 3431 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.004307 3428 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:16.013612 3416 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.014300 3408 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:16.016323 3464 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.017999 3463 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.016443 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:16.020690 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 213393us
I20260927 19:38:16.029428 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.031184 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.027936 3334 raft_consensus.cc:1067] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:16.024284 3359 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.040512 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.032080 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.043387 3471 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.043551 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:16.047681 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.044476 3475 consensus_queue.cc:1046] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:16.039489 3428 raft_consensus.cc:1067] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:38:16.044772 3375 consensus_queue.cc:1046] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260927 19:38:16.032166 3465 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.049916 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.049189 3459 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.031303 3416 consensus_queue.cc:1046] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260927 19:38:16.061488 3359 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.038484 3368 consensus_queue.cc:1046] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260927 19:38:16.018596 3463 consensus_queue.cc:1046] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260927 19:38:16.229012 3414 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:16.054334 3478 consensus_queue.cc:1046] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20260927 19:38:16.060470 3464 consensus_queue.cc:1046] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
W20260927 19:38:16.230963 3332 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.561s user 0.010s sys 0.011s
W20260927 19:38:16.241792 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 162492us
W20260927 19:38:16.230767 3427 consensus_meta.cc:287] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.733s user 0.013s sys 0.005s
I20260927 19:38:16.251027 2687 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34940
I20260927 19:38:16.276443 2687 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:16.276791 2687 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:16.029840 3420 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.016s sys 0.013s
I20260927 19:38:16.227674 3471 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.267064 3405 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:16.261562 3390 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:16.234740 2682 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:16.065774 3454 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.272730 3426 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260927 19:38:16.311583 2687 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:16.285841 2110 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.018s sys 0.015s
W20260927 19:38:16.299944 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.010s sys 0.020s
I20260927 19:38:16.276185 3372 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:16.274616 3414 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:16.315836 3390 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:16.312892 3369 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.011s sys 0.030s
W20260927 19:38:16.285853 3420 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.881s user 0.016s sys 0.013s
W20260927 19:38:16.285567 3419 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.019s sys 0.055s
I20260927 19:38:16.281878 3427 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.312785 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:16.357669 2687 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:16.282591 3465 raft_consensus.cc:3032] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:16.384999 2034 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:16.308840 3409 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.000s sys 0.035s
I20260927 19:38:16.272753 3332 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.385507 3372 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:16.399238 3345 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:16.399628 3345 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:16.400267 3332 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:16.356179 3334 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:16.362655 2681 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.018s sys 0.056s
W20260927 19:38:16.362846 3294 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.007s sys 0.012s
W20260927 19:38:16.402974 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 48674.8us
I20260927 19:38:16.370266 3471 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:16.379302 2110 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.702s user 0.018s sys 0.023s
W20260927 19:38:16.386952 3419 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s user 0.020s sys 0.055s
I20260927 19:38:16.384274 3416 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:16.396703 3409 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s user 0.001s sys 0.035s
I20260927 19:38:16.387633 3466 consensus_queue.cc:1046] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260927 19:38:16.386346 3369 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.012s sys 0.032s
I20260927 19:38:16.386168 3414 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.399821 3372 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:16.379331 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s user 0.010s sys 0.020s
W20260927 19:38:16.378592 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.043s sys 0.072s
I20260927 19:38:16.313119 3405 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.395735 3465 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:16.390309 3454 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:16.392382 3478 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.385288 3368 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:16.389858 3427 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:16.420254 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 794263us
W20260927 19:38:16.407938 3294 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s user 0.008s sys 0.012s
W20260927 19:38:16.426471 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 764280us
I20260927 19:38:16.452420 2110 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:16.403273 2681 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.018s sys 0.056s
I20260927 19:38:16.387329 3390 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:16.429097 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.043s sys 0.072s
I20260927 19:38:16.408138 2136 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:16.400403 3345 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.428546 3441 raft_consensus.cc:3032] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:16.454972 2110 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:16.419838 3473 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:16.430310 3372 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.437021 3368 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:16.468375 3368 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.469554 3465 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.471891 2110 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:16.473248 2138 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:16.473584 3406 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:16.474246 3406 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:16.416462 3471 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.474748 2140 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:16.475710 3405 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.461475 3476 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.462018 3420 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.451807 3426 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:16.462456 3359 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.465183 3473 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:16.475332 3406 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.482376 3473 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.482296 3359 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.484333 3359 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:16.457780 3332 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.409971 3334 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.474360 3409 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.417641 3411 ts_tablet_manager.cc:1918] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:16.477115 2110 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:16.470007 3368 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.479357 3441 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:16.455950 3324 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.021s sys 0.021s
W20260927 19:38:16.489519 3421 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.021s sys 0.105s
I20260927 19:38:16.420735 3416 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.476172 3471 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:16.546805 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 73532.5us
I20260927 19:38:16.451614 3478 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.495396 3345 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.561676 3474 raft_consensus.cc:3032] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:16.563542 2110 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:16.550493 3324 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.021s sys 0.025s
I20260927 19:38:16.574030 3415 raft_consensus.cc:3032] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
W20260927 19:38:16.555950 3421 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.021s sys 0.106s
I20260927 19:38:16.498937 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.548447 3332 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:16.487207 3420 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.557297 3411 log.cc:1196] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/f19d4799cf134d51b543a2952f564adc
W20260927 19:38:16.550710 3432 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.005s sys 0.006s
I20260927 19:38:16.487483 3426 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.485378 3476 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.597231 3413 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.598021 2687 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:38:16.598346 3476 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:16.598565 2687 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:16.593883 3448 raft_consensus.cc:3032] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:16.598942 2687 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.599015 3448 raft_consensus.cc:729] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:16.576293 3474 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:16.588372 3409 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260927 19:38:16.594569 3432 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.005s sys 0.006s
I20260927 19:38:16.624898 2110 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:16.591148 3415 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:16.590742 3332 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:16.627633 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 581567us
I20260927 19:38:16.594029 2684 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:16.639895 2687 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.650257 2687 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:16.623163 3409 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:16.657199 2034 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:16.652753 1961 connection.cc:232] RPC call timeout handler was delayed by -1.0188468959999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260927 19:38:16.661531 1961 leader_election.cc:334] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.503s (SENT)
I20260927 19:38:16.662070 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.633008 3332 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.649508 3480 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:16.663136 3431 consensus_meta.cc:287] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.650s user 0.012s sys 0.003s
I20260927 19:38:16.651772 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:16.663749 3409 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.588927 3441 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.683138 2683 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:16.686444 3484 log.cc:924] Time spent T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260927 19:38:16.687525 2681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 246) took 1117 ms. Trace:
I20260927 19:38:16.687893 2681 rpcz_store.cc:276] 0927 19:38:15.570665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:15.579497 (+ 8832us) service_pool.cc:227] Handling call
0927 19:38:15.623343 (+ 43846us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:15.660799 (+ 37456us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:15.663832 (+ 3033us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:15.683840 (+ 20008us) log.cc:842] Serialized 26 byte log entry
0927 19:38:16.463143 (+779303us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:16.463193 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:16.463257 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:16.687275 (+224018us) raft_consensus.cc:1671] finished
0927 19:38:16.687304 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:16.687447 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14770,"raft.queue_time_us":5822,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":6900,"thread_start_us":771790,"threads_started":1,"wal-append.queue_time_us":864546}
I20260927 19:38:16.690851 3459 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:16.692772 2683 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:16.693120 3479 log.cc:924] Time spent T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20260927 19:38:16.694172 1959 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:16.696026 2653 leader_election.cc:334] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.658s (SENT)
I20260927 19:38:16.699966 2653 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.667762 3471 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:16.670163 3464 raft_consensus.cc:3032] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:16.646623 3473 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.706832 3411 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60280 (request call id 1176) took 2115 ms. Trace:
I20260927 19:38:16.707401 3411 rpcz_store.cc:276] 0927 19:38:14.591215 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:14.592015 (+ 800us) service_pool.cc:227] Handling call
0927 19:38:14.793259 (+201244us) ts_tablet_manager.cc:1145] Deleting tablet f19d4799cf134d51b543a2952f564adc
0927 19:38:14.799641 (+ 6382us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0927 19:38:15.996912 (+1197271us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:16.395117 (+398205us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:16.706699 (+311582us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":183073,"thread_start_us":129479,"threads_started":1}
I20260927 19:38:16.708810 3473 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.709964 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet f19d4799cf134d51b543a2952f564adc (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:16.680166 3375 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.600037 3448 consensus_queue.cc:260] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.720374 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:16.726711 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:16.727531 2143 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:16.673756 3415 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.697103 3427 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:16.701577 3471 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:16.668459 3474 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.606755 3413 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.705968 3464 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:16.670212 3482 consensus_queue.cc:1035] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260927 19:38:16.749322 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:16.753633 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.747762 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:16.755867 2136 raft_consensus.cc:3032] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:16.755995 3462 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:16.756246 2136 raft_consensus.cc:729] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:16.756671 3413 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:16.756990 3489 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260927 19:38:16.757215 2136 consensus_queue.cc:260] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.757786 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:16.757649 3464 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.758474 2136 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.758797 3464 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.759548 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:16.760545 2143 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:16.761709 3437 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.762506 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:16.764067 2037 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.027s sys 0.061s
W20260927 19:38:16.764834 2037 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.027s sys 0.061s
I20260927 19:38:16.765839 3462 raft_consensus.cc:686] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:16.765336 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 717727us
I20260927 19:38:16.735585 3441 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:16.731213 3478 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.736461 3416 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:16.724968 3491 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.744306 3471 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.780349 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.781580 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.783313 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.745208 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:16.785636 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.754870 3334 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.849713 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:16.804422 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 74336.6us
I20260927 19:38:16.743664 3427 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.817896 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.851672 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:16.815672 3485 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260927 19:38:16.772426 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:16.876173 1961 leader_election.cc:334] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.520s (SENT)
I20260927 19:38:16.784523 3466 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.803121 3488 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.804952 3463 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.803463 3474 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.876686 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:16.874095 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:16.908598 1880 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3d1bc2cd9efe43399ddba396f5ddff85 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:16.877830 2141 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:16.784924 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:16.913504 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.852869 3431 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:16.803326 3473 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.766750 3462 consensus_queue.cc:237] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:16.874667 3359 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.859731 2137 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:16.784328 3413 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260927 19:38:16.911298 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:16.911607 1961 leader_election.cc:334] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.553s (SENT)
I20260927 19:38:16.915050 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.914472 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:16.931464 2140 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:16.862651 2683 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:16.816041 3420 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.868770 3426 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:16.817435 3491 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260927 19:38:16.948864 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.935060 3466 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.929745 3463 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.957966 1959 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.958484 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:16.966830 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:16.950908 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.938386 3488 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:16.968703 2110 leader_election.cc:334] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.044s (SENT)
I20260927 19:38:16.969367 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:16.970191 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:16.971897 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:16.972023 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:16.972370 2067 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:16.972380 2063 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:16.970964 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:16.962688 3499 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.109s
I20260927 19:38:16.957473 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:16.960706 3359 consensus_queue.cc:1046] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
W20260927 19:38:17.165673 3477 log.cc:924] Time spent T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.695s user 0.000s sys 0.003s
I20260927 19:38:16.973754 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:17.158190 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:17.016331 3409 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.076514 3465 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:17.053414 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:17.135298 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:16.974136 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.215019 2068 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
W20260927 19:38:17.046808 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 135213us
W20260927 19:38:17.216684 1961 leader_election.cc:334] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.568s (SENT)
I20260927 19:38:16.974203 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.217974 2071 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:17.218606 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.219589 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:16.974278 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.220070 2071 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:17.220811 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.221463 2144 raft_consensus.cc:1799] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Handling vote request from an unknown peer 4cc2df048f424e08803a390946004df0
I20260927 19:38:16.957607 3416 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.047371 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.144811 3421 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.084950 3482 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.148171 3480 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:17.148340 3441 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.970386 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.226075 3480 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.226869 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.227937 2065 raft_consensus.cc:2362] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
I20260927 19:38:17.228986 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.074030 1993 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:17.062281 3372 consensus_meta.cc:287] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.597s user 0.009s sys 0.003s
I20260927 19:38:17.224810 3464 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.009449 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.106956 3463 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:17.202740 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:17.202582 3332 consensus_meta.cc:287] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.533s user 0.010s sys 0.000s
I20260927 19:38:17.265902 3463 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.266193 3332 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.267792 3431 raft_consensus.cc:3032] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:17.268173 3431 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:17.268313 3332 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:17.203615 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.242585 3480 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.269339 2143 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:17.269008 3431 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.270911 3431 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.271011 2071 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:17.109090 2653 leader_election.cc:334] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.520s (SENT)
I20260927 19:38:17.272493 2653 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:17.273313 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 164265us
W20260927 19:38:17.273976 2653 leader_election.cc:334] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.547s (SENT)
I20260927 19:38:17.274293 2653 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:17.208482 2070 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:17.046497 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:16.989595 3368 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:16.974529 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.229897 3421 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.279610 3372 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.280158 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.280391 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.280807 2062 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:17.212358 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 243657us
W20260927 19:38:17.281739 2110 leader_election.cc:334] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.044s (SENT)
I20260927 19:38:17.272862 3462 consensus_queue.cc:1046] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:17.282194 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:17.282932 3368 raft_consensus.cc:3032] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:17.283314 3368 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:17.020993 3345 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:16.978674 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:17.074607 2136 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:17.009034 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.284121 3368 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.284720 2069 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:17.285080 3368 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.275748 2063 raft_consensus.cc:2362] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
I20260927 19:38:17.286012 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.221882 2144 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:17.209805 3414 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.015s sys 0.012s
I20260927 19:38:16.989945 3426 consensus_queue.cc:1046] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.221s
I20260927 19:38:17.223721 2140 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
W20260927 19:38:17.224169 1961 leader_election.cc:334] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.578s (SENT)
I20260927 19:38:17.125566 3427 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.209043 3454 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.231228 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.229053 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.214387 3494 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:17.122409 3471 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.200376 3334 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:17.296425 3494 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.296810 3334 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:17.297778 3334 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.298439 3494 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:17.298777 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.298995 3334 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.299320 2063 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.299998 3409 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:17.232965 2065 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.227829 3441 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.286116 3473 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.301265 3473 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.301992 3471 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.302380 3441 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.242313 1996 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
W20260927 19:38:17.288357 3414 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s user 0.015s sys 0.012s
I20260927 19:38:17.303455 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.238526 1959 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:16.973464 3466 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.228840 3482 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:16.974179 3493 consensus_queue.cc:1046] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.266s
I20260927 19:38:17.260491 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.230304 2683 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:17.275964 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.277407 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.285172 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.285517 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.269045 1995 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:17.042573 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.313643 2071 raft_consensus.cc:2354] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3.
I20260927 19:38:17.321193 2034 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:17.321616 1994 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:17.316138 1987 raft_consensus.cc:2396] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:17.316555 3482 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:17.291642 2068 raft_consensus.cc:2362] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
I20260927 19:38:17.328035 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.328572 2068 raft_consensus.cc:2362] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
I20260927 19:38:17.329299 2062 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:17.289728 1961 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:17.329294 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.330230 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:17.330339 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 106205us
I20260927 19:38:17.331681 3475 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:17.319008 3415 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.294484 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.330341 2068 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:17.293277 2654 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.288955 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.296716 3332 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:16.990669 3420 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.304546 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:17.295933 2037 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:48570 dropped due to backpressure. The service queue is full; it has 50 items.
W20260927 19:38:17.306543 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 68111.6us
I20260927 19:38:17.308432 1989 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:17.308606 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:17.309227 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.307657 3473 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.354049 2687 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:17.310009 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.294025 3345 raft_consensus.cc:3032] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:17.311793 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.312639 2066 raft_consensus.cc:2379] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:17.313031 2063 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:17.307427 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:17.316843 2070 raft_consensus.cc:2354] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3.
I20260927 19:38:17.318143 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.319504 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.290136 3421 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.290283 3413 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.261758 3464 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.288210 3359 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.318733 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.317070 2143 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:17.325093 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:17.326759 2065 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:17.294368 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.338270 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.313417 3414 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:17.314924 3493 raft_consensus.cc:3032] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:17.353605 2064 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:17.398774 2681 raft_consensus.cc:2379] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:17.341884 1996 raft_consensus.cc:2379] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:17.367065 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.290086 3463 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.379541 2067 raft_consensus.cc:2362] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
I20260927 19:38:17.316968 1990 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:17.341882 3437 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:17.398305 3493 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:17.384179 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45988.1us
I20260927 19:38:17.385532 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.385699 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.416373 2070 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.382992 1995 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:17.417583 3437 raft_consensus.cc:686] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:17.418377 1995 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:17.419330 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.315163 3466 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.385862 3471 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:17.394213 3416 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:17.374576 3345 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:17.394048 3359 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.400790 2037 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:48570 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1205)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 99)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 87)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 83)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1653)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1654)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1655)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1657)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 85)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 89)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 91)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 92)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 95)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 96)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 103)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1188)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1189)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1190)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1192)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1193)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1195)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1196)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1197)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1199)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1208)
I20260927 19:38:17.342036 3332 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.423092 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.465473 3493 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.524101 3345 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.525121 3493 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.440276 1994 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260927 19:38:17.412106 1996 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:17.418576 3437 consensus_queue.cc:237] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.408433 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.546077 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.546278 3471 raft_consensus.cc:686] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:17.547991 2063 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:17.605209 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 187751us
I20260927 19:38:17.381585 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.606763 2068 raft_consensus.cc:2362] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
I20260927 19:38:17.616539 2069 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:17.619843 2070 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:17.420843 1987 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:17.623955 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.624673 2069 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:17.625636 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.341806 2062 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.622257 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.627166 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.627239 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.532960 3416 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:17.628938 1959 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:17.632114 2070 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:17.639602 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.461613 3448 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.618777 3466 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.644228 2068 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:17.621493 3345 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.650007 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.619362 3464 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.645340 3416 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:17.661863 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.662405 2071 raft_consensus.cc:2362] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2. Re-sending same reply.
I20260927 19:38:17.663066 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.663622 2071 raft_consensus.cc:2379] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:17.647331 3427 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:17.642612 2063 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:17.622902 3463 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:17.653211 2068 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:17.621493 3476 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:17.646649 2070 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:17.548012 3471 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.668355 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.680121 2064 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:17.623328 2037 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:48542 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1653)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1186)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1187)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1654)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1655)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1659)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1657)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 85)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1202)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1661)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1660)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1662)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1664)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 89)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 91)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 92)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 95)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 96)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 100)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 103)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1185)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1188)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1189)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1190)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1191)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1192)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1193)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1194)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1195)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1196)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1197)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1198)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1199)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1200)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1208)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1670)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1673)
I20260927 19:38:17.681118 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.682067 2064 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:17.683028 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.683758 2064 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:17.684572 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.685279 2064 raft_consensus.cc:2354] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3.
I20260927 19:38:17.669045 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.670742 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.672075 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.686244 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.669224 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.686739 2068 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
W20260927 19:38:17.676184 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 345112us
W20260927 19:38:17.687597 1961 leader_election.cc:334] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.587s (SENT)
I20260927 19:38:17.687059 2063 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:17.687793 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.688119 1961 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:17.688664 2071 raft_consensus.cc:2379] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:17.686841 2064 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.678714 3463 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.692565 2067 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:17.694314 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:17.700116 2653 leader_election.cc:334] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.729s (SENT)
I20260927 19:38:17.700591 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.700783 2653 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:17.701674 2070 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:17.702833 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.703603 2070 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:17.704555 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.704964 2110 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.705237 2070 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:17.706437 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 87807.3us
I20260927 19:38:17.710054 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.711169 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:17.712445 1961 leader_election.cc:334] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.602s (SENT)
W20260927 19:38:17.713703 1961 leader_election.cc:334] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.617s (SENT)
I20260927 19:38:17.717844 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.718098 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.719080 2068 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:17.719233 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.723181 2110 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.669595 3359 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.727317 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.725494 2064 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
W20260927 19:38:17.730437 1961 leader_election.cc:334] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.620s (SENT)
I20260927 19:38:17.730612 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.654652 3466 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.727277 3421 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:17.676363 3497 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.729225 2068 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:17.726440 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.741539 2653 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:17.748347 2070 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
W20260927 19:38:17.737794 1961 leader_election.cc:334] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.665s (SENT)
I20260927 19:38:17.753944 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.731731 2064 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:17.757526 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:17.757516 2037 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:17.758352 1992 raft_consensus.cc:1789] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.3
I20260927 19:38:17.721242 3491 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.761080 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.720777 3454 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:17.761117 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.759750 1992 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:17.727924 3426 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.765537 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.760277 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:17.739409 3432 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:17.691232 3476 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.758086 3409 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:17.732245 3421 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:17.778615 2070 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:17.786477 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:17.763091 3464 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.760582 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:17.779743 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.790047 2068 raft_consensus.cc:2354] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3.
I20260927 19:38:17.760666 3359 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:17.794261 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:17.765674 3332 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.769047 3473 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:17.775799 3491 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:17.797789 2067 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:17.802017 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:17.809561 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:17.810204 2068 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
W20260927 19:38:17.802659 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 58468.2us
W20260927 19:38:17.805546 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 59102.8us
I20260927 19:38:17.810941 2110 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:17.807695 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 66388.8us
W20260927 19:38:17.796144 3465 consensus_meta.cc:287] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.582s user 0.016s sys 0.027s
I20260927 19:38:17.813102 2110 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.810979 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.814061 2068 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.815382 3482 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:17.815987 3482 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:17.816903 3482 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.820838 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:17.821568 2687 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:17.803871 3359 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.785599 3432 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:17.826422 3476 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.792502 3421 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:17.795274 2679 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260927 19:38:17.800221 3334 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.501s user 0.008s sys 0.012s
I20260927 19:38:17.797528 3464 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:17.794773 3420 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:17.828450 2108 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:17.804329 3409 raft_consensus.cc:686] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:17.811580 3491 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:17.812418 2682 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:17.827680 3473 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:17.838508 3428 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:17.844908 2071 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:17.884672 3519 log.cc:924] Time spent T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260927 19:38:17.826998 3432 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:17.888870 3359 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.834386 3406 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.838670 3332 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.890688 3466 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.894615 3463 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:17.880326 3491 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:17.897598 3524 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:17.898161 3523 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:17.834053 3454 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:17.920053 3416 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:17.921717 3409 consensus_queue.cc:237] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:17.925295 3368 consensus_meta.cc:287] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.633s user 0.031s sys 0.039s
W20260927 19:38:17.930845 3415 consensus_meta.cc:287] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.598s user 0.013s sys 0.014s
W20260927 19:38:17.914466 3431 consensus_meta.cc:287] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.643s user 0.025s sys 0.046s
I20260927 19:38:17.935330 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:17.935153 3421 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.175107 2063 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
W20260927 19:38:18.178062 2062 consensus_meta.cc:287] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.527s user 0.013s sys 0.000s
I20260927 19:38:18.190472 2062 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:18.191039 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:18.178576 3406 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.191386 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.191695 2065 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:18.192026 2062 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:18.192920 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.193567 2062 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:18.178778 3332 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.194818 2062 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:18.180455 3524 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.195904 3524 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:18.182019 3454 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.183517 3416 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.180673 3523 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.185447 2685 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:18.184023 2067 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:18.185469 3421 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.202879 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:18.213909 3332 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:18.214879 1990 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1163) took 1476 ms (client timeout 1720 ms). Trace:
I20260927 19:38:18.215394 3523 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:18.215529 1990 rpcz_store.cc:269] 0927 19:38:16.738303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:16.762200 (+ 23897us) service_pool.cc:227] Handling call
0927 19:38:18.214587 (+1452387us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b540031c254. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:18.214762 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724245760}
I20260927 19:38:18.193174 3406 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:17.936646 3428 raft_consensus.cc:686] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:18.218003 2065 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:18.109614 3482 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:18.218603 3482 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.219283 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 131) took 1486 ms. Trace:
I20260927 19:38:18.219666 1994 rpcz_store.cc:276] 0927 19:38:16.733032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.372195 (+639163us) service_pool.cc:227] Handling call
0927 19:38:17.421473 (+ 49278us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:17.528874 (+107401us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.219093 (+690219us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b54003606d0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:18.219225 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37952384}
I20260927 19:38:18.220646 3482 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:18.110319 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 281954us
I20260927 19:38:18.222434 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 129) took 1499 ms. Trace:
I20260927 19:38:18.222838 1987 rpcz_store.cc:276] 0927 19:38:16.723070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.367828 (+644758us) service_pool.cc:227] Handling call
0927 19:38:17.404660 (+ 36832us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:17.633095 (+228435us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.222137 (+589042us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b540020e520. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138f2730 00007f7913965b42 00007f7913961f26 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:38:18.222375 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63366912}
I20260927 19:38:18.223987 3406 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:18.227989 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 101) took 1069 ms. Trace:
I20260927 19:38:18.231683 2064 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:18.228374 2065 rpcz_store.cc:276] 0927 19:38:17.158985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.190742 (+1031757us) service_pool.cc:227] Handling call
0927 19:38:18.227923 (+ 37181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.201084 3491 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.247754 3491 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:18.257733 1879 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 60943.5us
I20260927 19:38:18.217558 3428 consensus_queue.cc:237] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.108930 3532 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:18.237133 2067 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:18.269767 3532 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.272553 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 91) took 1149 ms. Trace:
I20260927 19:38:18.239982 3524 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.280205 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 92) took 1156 ms. Trace:
I20260927 19:38:18.284283 2067 rpcz_store.cc:276] 0927 19:38:17.123477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.273528 (+1150051us) service_pool.cc:227] Handling call
0927 19:38:18.273847 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.274013 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.274040 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.274079 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:18.274110 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.274160 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.274249 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:18.289857 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 552059us
I20260927 19:38:18.277300 2065 rpcz_store.cc:276] 0927 19:38:17.123159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.241647 (+1118488us) service_pool.cc:227] Handling call
0927 19:38:18.242070 (+ 423us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:18.259839 (+ 17769us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.259891 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:18.260461 (+ 570us) log.cc:842] Serialized 26 byte log entry
0927 19:38:18.265964 (+ 5503us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:18.271660 (+ 5696us) log.cc:842] Serialized 14 byte log entry
0927 19:38:18.272171 (+ 511us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.272253 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:18.272323 (+ 70us) raft_consensus.cc:1671] finished
0927 19:38:18.272337 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.272495 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":5544,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":2212,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":207,"spinlock_wait_cycles":33664,"thread_start_us":10218,"threads_started":2,"wal-append.queue_time_us":5433}
W20260927 19:38:18.290537 1961 leader_election.cc:334] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.697s (SENT)
I20260927 19:38:18.290941 1961 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.291548 2065 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:18.291576 3415 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:18.291854 1961 leader_election.cc:334] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.782s (SENT)
I20260927 19:38:18.292269 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.292366 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 96) took 1138 ms. Trace:
I20260927 19:38:18.195369 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 89) took 1073 ms. Trace:
I20260927 19:38:18.292975 3415 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:18.293135 2062 rpcz_store.cc:276] 0927 19:38:17.122407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.194225 (+1071818us) service_pool.cc:227] Handling call
0927 19:38:18.194514 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.195142 (+ 628us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.195323 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.288280 3524 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:18.288720 3532 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:18.292796 2065 rpcz_store.cc:276] 0927 19:38:17.153591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.290845 (+1137254us) service_pool.cc:227] Handling call
0927 19:38:18.291132 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.292093 (+ 961us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.292301 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.295984 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 100) took 1137 ms. Trace:
I20260927 19:38:18.290114 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 95) took 1142 ms. Trace:
I20260927 19:38:18.297032 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 103) took 1137 ms. Trace:
W20260927 19:38:18.293244 1961 leader_election.cc:334] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.820s (SENT)
I20260927 19:38:18.298251 1961 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:18.298961 3415 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:18.299921 1961 leader_election.cc:334] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.856s (SENT)
W20260927 19:38:18.301069 1961 leader_election.cc:334] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.862s (SENT)
W20260927 19:38:18.301983 1961 leader_election.cc:334] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.876s (SENT)
I20260927 19:38:18.304605 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:18.297596 2065 rpcz_store.cc:276] 0927 19:38:17.159482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.295870 (+1136388us) service_pool.cc:227] Handling call
0927 19:38:18.296148 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.296686 (+ 538us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.296722 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.296767 (+ 45us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:18.296800 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.296862 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.296983 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69}
I20260927 19:38:18.305516 3415 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:18.306420 1961 leader_election.cc:334] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.886s (SENT)
W20260927 19:38:18.302363 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 500405us
I20260927 19:38:18.310658 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.312089 3431 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:18.313257 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.314005 3431 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:18.315145 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.315832 3431 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:18.306798 1961 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.317587 2110 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.317909 3415 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:18.319026 1961 leader_election.cc:334] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.893s (SENT)
W20260927 19:38:18.319813 1961 leader_election.cc:334] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.921s (SENT)
W20260927 19:38:18.320606 1961 leader_election.cc:334] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.929s (SENT)
I20260927 19:38:18.320835 3532 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:18.320960 1961 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:18.321514 3532 raft_consensus.cc:686] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:18.322448 3532 consensus_queue.cc:237] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.299629 2067 rpcz_store.cc:276] 0927 19:38:17.147834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.289288 (+1141454us) service_pool.cc:227] Handling call
0927 19:38:18.289632 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.289804 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.289835 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.289877 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:18.289909 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.289963 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.290062 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.296921 2062 rpcz_store.cc:276] 0927 19:38:17.158628 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.294202 (+1135574us) service_pool.cc:227] Handling call
0927 19:38:18.294511 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.295605 (+ 1094us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.295643 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.295698 (+ 55us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:18.295734 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.295803 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.295927 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":600,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107}
I20260927 19:38:18.326582 2653 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:18.328647 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:18.329062 3491 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:18.329497 3491 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:18.329861 3491 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:18.317822 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.332116 2069 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:18.332568 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1652) took 1054 ms. Trace:
I20260927 19:38:18.332933 2069 rpcz_store.cc:276] 0927 19:38:17.277698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.317546 (+1039848us) service_pool.cc:227] Handling call
0927 19:38:18.332516 (+ 14970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.333710 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.325425 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1189) took 1130 ms. Trace:
I20260927 19:38:18.334219 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1191) took 1138 ms. Trace:
I20260927 19:38:18.334318 1994 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:18.334631 2069 rpcz_store.cc:276] 0927 19:38:17.195884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.333553 (+1137669us) service_pool.cc:227] Handling call
0927 19:38:18.333798 (+ 245us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.333959 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.333982 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.334018 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.334048 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.334092 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.334178 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.335120 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.335772 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.336261 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1192) took 1140 ms. Trace:
I20260927 19:38:18.336452 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.335768 1990 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:18.337159 1987 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:18.337728 3524 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:18.339353 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.418314 1987 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:18.343102 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.419693 1994 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:17.840235 2068 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:18.421461 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1194) took 1224 ms. Trace:
I20260927 19:38:18.344890 2685 pending_rounds.cc:76] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Aborting all ops after (but not including) 4
I20260927 19:38:18.422196 2685 pending_rounds.cc:97] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260927 19:38:18.421860 2068 rpcz_store.cc:276] 0927 19:38:17.196742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.420589 (+1223847us) service_pool.cc:227] Handling call
0927 19:38:18.420931 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.421126 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.421155 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.421199 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.421232 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.421288 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.421408 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.422780 2685 raft_consensus.cc:2866] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:18.424134 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1195) took 1227 ms. Trace:
I20260927 19:38:18.424494 2068 rpcz_store.cc:276] 0927 19:38:17.197009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.423256 (+1226247us) service_pool.cc:227] Handling call
0927 19:38:18.423606 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.423779 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.423812 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.423868 (+ 56us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.423911 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.423974 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.424087 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.427591 2068 raft_consensus.cc:1226] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.428325 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1196) took 1231 ms. Trace:
I20260927 19:38:18.428694 2068 rpcz_store.cc:276] 0927 19:38:17.197279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.426982 (+1229703us) service_pool.cc:227] Handling call
0927 19:38:18.427370 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.428048 (+ 678us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.428265 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.345757 2680 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:18.348395 1991 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:18.336628 2069 rpcz_store.cc:276] 0927 19:38:17.196160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.335543 (+1139383us) service_pool.cc:227] Handling call
0927 19:38:18.335842 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.335995 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.336018 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.336054 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.336080 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.336129 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.336218 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.334254 2067 rpcz_store.cc:276] 0927 19:38:17.195273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.324567 (+1129294us) service_pool.cc:227] Handling call
0927 19:38:18.324872 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.325029 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.325059 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.325098 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.325129 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.325184 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.325290 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:18.369699 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 49109.5us
W20260927 19:38:18.465538 1961 leader_election.cc:334] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.943s (SENT)
I20260927 19:38:18.465968 1961 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.399555 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1193) took 1203 ms. Trace:
I20260927 19:38:18.400707 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.443688 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1197) took 1245 ms. Trace:
I20260927 19:38:18.467571 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:18.468140 2068 rpcz_store.cc:276] 0927 19:38:17.198378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.442552 (+1244174us) service_pool.cc:227] Handling call
0927 19:38:18.442999 (+ 447us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.443249 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.443285 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.443352 (+ 67us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.443394 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.443463 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.443616 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.450197 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.469413 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.459261 2140 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:18.460026 3448 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:18.461323 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:18.473945 3448 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.462822 2680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34930 (request call id 328) took 1137 ms. Trace:
I20260927 19:38:18.475096 2680 rpcz_store.cc:276] 0927 19:38:17.325762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.353661 (+ 27899us) service_pool.cc:227] Handling call
0927 19:38:18.462609 (+1108948us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54005ab254. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:18.462752 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1834513792}
I20260927 19:38:18.475893 3448 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.476418 3448 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:18.463841 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:18.472031 2110 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.466773 2071 rpcz_store.cc:276] 0927 19:38:17.196457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.398647 (+1202190us) service_pool.cc:227] Handling call
0927 19:38:18.399057 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.399241 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.399269 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.399311 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.399342 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.399394 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.399499 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.479761 2110 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.479925 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:18.481133 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1198) took 1282 ms. Trace:
I20260927 19:38:18.481663 2071 rpcz_store.cc:276] 0927 19:38:17.199501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.480131 (+1280630us) service_pool.cc:227] Handling call
0927 19:38:18.480516 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.480721 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.480758 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.480814 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:18.480867 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.480937 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.481067 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.472528 3493 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:18.470234 2069 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:18.488116 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.488206 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:18.489104 2068 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:18.489660 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:18.490787 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:18.490934 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1208) took 1284 ms. Trace:
I20260927 19:38:18.491796 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:18.501174 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1200) took 1301 ms. Trace:
I20260927 19:38:18.501816 2069 rpcz_store.cc:276] 0927 19:38:17.200096 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.486183 (+1286087us) service_pool.cc:227] Handling call
0927 19:38:18.500509 (+ 14326us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.500690 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.500724 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.500779 (+ 55us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.500821 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.500903 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.501101 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.491478 2068 rpcz_store.cc:276] 0927 19:38:17.206423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.489974 (+1283551us) service_pool.cc:227] Handling call
0927 19:38:18.490323 (+ 349us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.490524 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.490562 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.490622 (+ 60us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:18.490666 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.490735 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.490869 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.517472 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.476862 3448 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.529547 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1199) took 1285 ms. Trace:
I20260927 19:38:18.363093 2136 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.559721 2071 rpcz_store.cc:276] 0927 19:38:17.199794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.482870 (+1283076us) service_pool.cc:227] Handling call
0927 19:38:18.483229 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.483429 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.483465 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.484666 (+ 1201us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.484728 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.484810 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.485491 (+ 681us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:18.584758 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 79794.4us
I20260927 19:38:18.392242 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.590401 2069 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 19:38:18.591773 2071 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.592613 2068 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.505726 3480 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:18.544322 3416 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:18.593532 3482 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
W20260927 19:38:18.592998 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 120651us
I20260927 19:38:18.594395 2067 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.635279 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1671) took 1023 ms. Trace:
I20260927 19:38:18.635725 2067 rpcz_store.cc:276] 0927 19:38:17.612006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.586305 (+974299us) service_pool.cc:227] Handling call
0927 19:38:18.589851 (+ 3546us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.634974 (+ 45123us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.635209 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.597875 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.598287 2068 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.641690 2067 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.603376 3527 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:18.645929 3527 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.629029 3421 raft_consensus.cc:465] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Not starting pre-election -- already a leader
W20260927 19:38:18.649072 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 305085us
I20260927 19:38:18.594120 3359 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:18.650345 3359 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 19:38:18.634156 1961 leader_election.cc:334] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.956s (SENT)
I20260927 19:38:18.650893 3359 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.651398 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1188) took 1456 ms. Trace:
I20260927 19:38:18.653810 1961 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:18.654281 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54969.5us
I20260927 19:38:18.657234 2068 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:18.650454 2653 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.654336 2067 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.626415 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1673) took 1004 ms. Trace:
I20260927 19:38:18.659627 2071 rpcz_store.cc:276] 0927 19:38:17.622067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.591059 (+968992us) service_pool.cc:227] Handling call
0927 19:38:18.591547 (+ 488us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.625736 (+ 34189us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.626334 (+ 598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:18.660357 2067 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:18.593823 2140 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:18.664461 2110 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.667048 2067 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260927 19:38:18.667399 1961 leader_election.cc:334] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.023s (SENT)
I20260927 19:38:18.667842 1961 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.606783 3497 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:18.619750 3480 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.672614 2068 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:18.648929 3527 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:18.602598 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:18.679623 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 293696us
I20260927 19:38:18.682456 2110 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:18.621217 3545 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:18.651420 3359 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:18.687656 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.692354 2068 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260927 19:38:18.692499 1961 connection.cc:232] RPC call timeout handler was delayed by -1.7987767509999912s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
W20260927 19:38:18.692858 1961 connection.cc:232] RPC call timeout handler was delayed by -1.7600767510000423s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260927 19:38:18.693105 1961 connection.cc:232] RPC call timeout handler was delayed by -1.7519767510000293s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
I20260927 19:38:18.693459 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:18.693799 1961 connection.cc:232] RPC call timeout handler was delayed by -1.7015767510000614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
I20260927 19:38:18.694168 2068 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:18.655745 3482 raft_consensus.cc:686] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:18.694317 1961 connection.cc:232] RPC call timeout handler was delayed by -1.6547767509999858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
I20260927 19:38:18.670249 3421 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:18.705693 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:18.635407 3416 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.712657 3421 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.721313 2067 raft_consensus.cc:2379] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
W20260927 19:38:18.721009 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47224us
I20260927 19:38:18.721946 3550 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:18.723491 3550 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:18.720503 2062 raft_consensus.cc:2932] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:18.684425 3480 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:18.694666 3546 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:18.729930 3553 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:18.694540 3545 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.672231 3334 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:18.744726 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.726722 3421 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:18.731966 3546 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.740264 3553 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.740707 3542 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:18.743629 3545 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:18.770082 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 72196.6us
I20260927 19:38:18.755708 3546 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:18.765988 3553 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:18.761013 3448 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.767428 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:18.791800 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 107240us
I20260927 19:38:18.764544 3490 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
W20260927 19:38:18.797035 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 52434.8us
W20260927 19:38:18.804179 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 73351.8us
I20260927 19:38:18.651851 2065 rpcz_store.cc:276] 0927 19:38:17.194954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.305931 (+1110977us) service_pool.cc:227] Handling call
0927 19:38:18.306270 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:18.306638 (+ 368us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.306671 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:18.307187 (+ 516us) log.cc:842] Serialized 26 byte log entry
0927 19:38:18.472969 (+165782us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:18.473694 (+ 725us) log.cc:842] Serialized 14 byte log entry
0927 19:38:18.473872 (+ 178us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.473953 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:18.651119 (+177166us) raft_consensus.cc:1671] finished
0927 19:38:18.651163 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:18.651324 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":173,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":6640,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":57409,"thread_start_us":162283,"threads_started":1,"wal-append.queue_time_us":282920}
I20260927 19:38:18.797935 3448 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:18.806059 3497 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:18.795699 3558 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 19:38:18.854498 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 49411.9us
I20260927 19:38:18.779776 3482 consensus_queue.cc:237] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.863706 3559 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:18.866657 3416 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.817972 3448 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:18.822244 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:18.825111 3558 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.873104 3551 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:18.873641 3551 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.875273 3551 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.884557 3497 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:18.890404 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:18.891119 1996 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 3
I20260927 19:38:18.903431 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:18.866984 3559 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:18.904306 1995 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:18.870774 3448 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:18.914142 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:18.870107 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:18.948798 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:18.871564 3558 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:18.947016 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 145762us
I20260927 19:38:18.825845 3542 consensus_queue.cc:1046] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260927 19:38:18.958330 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:18.923442 3559 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:18.964864 1996 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:18.976496 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:18.978287 2687 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:18.965798 1961 leader_election.cc:334] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.617s (SENT)
I20260927 19:38:18.979773 1961 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:18.974989 2654 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:18.986217 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:18.941342 3464 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260927 19:38:18.952560 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:18.941583 3556 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260927 19:38:18.998219 2687 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:18.972950 3372 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:19.007853 3372 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:18.957516 2065 raft_consensus.cc:2362] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2. Re-sending same reply.
W20260927 19:38:18.983171 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1157) took 2257 ms (client timeout 1545 ms). Trace:
I20260927 19:38:18.987890 3554 consensus_queue.cc:1046] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260927 19:38:19.018143 3372 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:19.005125 2136 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:19.019387 3576 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:19.058739 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 203260us
I20260927 19:38:19.010670 3550 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.059316 3372 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:19.056612 3416 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260927 19:38:19.067094 3577 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:19.020663 1993 rpcz_store.cc:269] 0927 19:38:16.700180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:16.757535 (+ 57355us) service_pool.cc:227] Handling call
0927 19:38:18.950588 (+2193053us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b5400303ed4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:18.957167 (+ 6579us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":137264000}
I20260927 19:38:19.079300 3372 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.080651 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:19.085170 3576 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:19.081900 3542 raft_consensus.cc:3032] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:19.094411 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:19.089769 3372 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:19.095062 2067 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:19.056994 3552 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.068615 3550 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.088164 3565 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 19:38:19.176836 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 50258.4us
I20260927 19:38:19.179706 2068 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:19.151185 2110 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:19.132031 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 145880us
W20260927 19:38:19.205760 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54689.9us
I20260927 19:38:19.206485 2108 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:19.212710 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 246904us
W20260927 19:38:19.217299 1961 leader_election.cc:334] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.959s (SENT)
I20260927 19:38:19.218168 1961 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.211215 3565 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.088629 3570 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:19.235340 1961 leader_election.cc:334] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.983s (SENT)
I20260927 19:38:19.236466 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.157815 3579 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:19.180233 3575 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260927 19:38:19.124995 3584 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:19.180645 3553 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.243567 3584 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.097111 3542 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:19.220938 3497 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:19.211851 3413 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.238876 3564 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:19.261891 2068 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:19.157474 3566 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.090200 3571 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:19.223922 3334 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260927 19:38:19.097159 3480 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:19.237464 1961 leader_election.cc:334] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.023s (SENT)
I20260927 19:38:19.253480 3553 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:19.276767 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.281276 3580 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.282465 3580 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.285125 3580 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.259946 3564 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:19.300683 3587 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.301487 3587 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.304281 3587 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.263760 3497 raft_consensus.cc:686] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:19.278494 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:19.246655 3552 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.240723 3570 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.296756 3571 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:19.264151 3413 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.306289 3564 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:19.307101 1991 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1160) took 2594 ms (client timeout 2170 ms). Trace:
I20260927 19:38:19.089455 3577 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:19.363284 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 123837us
I20260927 19:38:19.321580 3587 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:19.241453 3579 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.280525 3546 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.317893 3542 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.440234 3542 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:19.445868 3334 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.446600 3334 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:19.447948 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee040 after lost signal to thread 818
W20260927 19:38:19.448385 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee060 after lost signal to thread 821
W20260927 19:38:19.448830 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee160 after lost signal to thread 1880
W20260927 19:38:19.452957 3591 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20260927 19:38:19.447005 3552 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.256886 3565 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.410598 3550 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:19.473899 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 233570us
I20260927 19:38:19.449126 3334 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.437558 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:19.363034 1991 rpcz_store.cc:269] 0927 19:38:16.711131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.202403 (+491272us) service_pool.cc:227] Handling call
0927 19:38:19.295792 (+2093389us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000e7ed4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:19.304933 (+ 9141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2431102336}
W20260927 19:38:19.491780 3359 consensus_meta.cc:287] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.778s user 0.011s sys 0.005s
I20260927 19:38:19.429327 3553 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
I20260927 19:38:19.477694 3587 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:19.497922 3558 consensus_meta.cc:287] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.548s user 0.017s sys 0.000s
I20260927 19:38:19.453583 3571 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.275511 3566 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:19.543704 3448 consensus_meta.cc:287] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.579s user 0.015s sys 0.000s
I20260927 19:38:19.538053 3573 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:19.460301 3497 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:19.560418 2062 consensus_meta.cc:287] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.811s user 0.013s sys 0.000s
I20260927 19:38:19.581161 2140 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260927 19:38:19.546239 3480 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:19.552652 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 943) took 1217 ms. Trace:
W20260927 19:38:19.576399 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 338698us
I20260927 19:38:19.539117 3359 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:19.584563 3583 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
W20260927 19:38:19.586313 3421 consensus_meta.cc:287] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.832s user 0.011s sys 0.001s
W20260927 19:38:19.596099 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 392564us
I20260927 19:38:19.587276 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:19.584489 3592 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:19.590754 3480 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.548017 3558 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.577597 3584 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.593390 3573 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.614823 3463 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.615053 3579 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.617627 3565 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.579281 3448 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.658536 3463 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.659206 3577 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.660625 3463 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.660476 3448 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.589143 2137 rpcz_store.cc:276] 0927 19:38:18.334801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.334950 (+ 149us) service_pool.cc:227] Handling call
0927 19:38:19.415834 (+1080884us) spinlock_profiling.cc:243] Waited 49.5 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:19.547825 (+131991us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113910528}
I20260927 19:38:19.596244 3413 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:19.632537 3596 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260927 19:38:19.657493 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 419996us
I20260927 19:38:19.609696 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:19.661149 3463 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:19.668790 1961 connection.cc:232] RPC call timeout handler was delayed by -1.5739427739999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
W20260927 19:38:19.669732 1961 connection.cc:232] RPC call timeout handler was delayed by -1.389442774000031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260927 19:38:19.670181 1961 connection.cc:232] RPC call timeout handler was delayed by -1.2688427740000634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
I20260927 19:38:19.670495 2067 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260927 19:38:19.663552 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:19.663226 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:19.672475 3548 log.cc:924] Time spent T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260927 19:38:19.613926 3480 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.664618 3589 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.662220 3448 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.666352 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1190) took 2461 ms. Trace:
I20260927 19:38:19.608668 3421 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.675354 3546 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:19.678850 3559 consensus_meta.cc:287] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.688s user 0.008s sys 0.016s
I20260927 19:38:19.695528 3566 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.702633 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:19.680307 2680 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:19.708179 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.708748 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 130) took 1858 ms. Trace:
I20260927 19:38:19.710130 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:19.697264 3545 consensus_meta.cc:287] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.919s user 0.009s sys 0.004s
I20260927 19:38:19.689261 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:19.706430 2138 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:19.711009 2685 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260927 19:38:19.707952 2654 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:19.707111 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 250836us
W20260927 19:38:19.713479 3593 log.cc:924] Time spent T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
I20260927 19:38:19.713550 3594 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:19.710336 2145 rpcz_store.cc:276] 0927 19:38:17.841312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.841879 (+ 567us) service_pool.cc:227] Handling call
0927 19:38:18.889757 (+1047878us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:18.889975 (+ 218us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.890174 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 19:38:18.995167 (+104993us) log.cc:842] Serialized 14 byte log entry
0927 19:38:19.699647 (+704480us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:19.699695 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:19.699733 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:19.699788 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:19.699894 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":175301,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":138135,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":19353,"spinlock_wait_cycles":191055232,"thread_start_us":750902,"threads_started":3,"wal-append.queue_time_us":472236}
I20260927 19:38:19.732432 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:19.718788 2034 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:19.733093 3418 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:19.733203 1996 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:19.729791 1961 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.732559 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:19.674810 3463 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.736197 1995 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:19.736178 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.736617 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:19.733916 3418 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.739287 3597 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.740154 3597 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.743577 3597 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.693027 3448 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:19.745131 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 118985us
I20260927 19:38:19.747665 3601 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260927 19:38:19.722095 3494 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:19.768163 1961 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.710561 3579 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260927 19:38:19.765413 2683 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:19.757225 3372 consensus_meta.cc:287] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.662s user 0.009s sys 0.003s
I20260927 19:38:19.769901 1961 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:19.744517 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:19.745776 3418 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:19.716372 3545 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.750070 3359 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:19.773918 2111 leader_election.cc:334] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.545s (SENT)
I20260927 19:38:19.706008 3559 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.776249 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:19.766261 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:19.768491 3597 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:19.782855 1961 leader_election.cc:398] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3.
I20260927 19:38:19.749462 3570 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.779474 3604 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:19.794126 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:19.794865 1996 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:19.784672 2686 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:19.786402 3573 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.786854 3585 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:19.790535 1995 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:19.791734 1961 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.777280 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:19.799535 1961 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.776682 3601 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:19.777871 3599 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:19.793208 1987 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:19.803858 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:19.789831 3597 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.749924 3594 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.808578 3597 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:19.812111 3600 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:19.810364 3599 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:19.812672 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:19.813079 3599 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:19.783847 1991 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:19.697434 3589 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:19.779914 3372 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.803193 3601 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:19.806767 3584 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:19.825346 1996 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:19.808156 3595 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:19.813589 3448 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.827487 2654 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.836151 2688 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:19.837792 1961 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:19.838128 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:19.842648 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:19.826874 3601 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:19.814536 3573 consensus_queue.cc:1046] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260927 19:38:19.694439 2062 rpcz_store.cc:276] 0927 19:38:17.195579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.327096 (+1131517us) service_pool.cc:227] Handling call
0927 19:38:18.327718 (+ 622us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:18.328132 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.328165 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:18.629410 (+301245us) log.cc:842] Serialized 501 byte log entry
0927 19:38:18.720346 (+ 90936us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:19.588018 (+867672us) log.cc:842] Serialized 14 byte log entry
0927 19:38:19.598039 (+ 10021us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:19.608794 (+ 10755us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:19.611401 (+ 2607us) raft_consensus.cc:1671] finished
0927 19:38:19.613435 (+ 2034us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:19.625393 (+ 11958us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":67830,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":125,"thread_start_us":90456,"threads_started":1,"wal-append.queue_time_us":98818}
I20260927 19:38:19.826534 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:19.829250 3480 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.840698 3448 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.837548 3584 raft_consensus.cc:686] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:19.835740 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:19.841615 3494 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:19.846472 3604 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:19.985481 1995 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260927 19:38:19.985786 2111 leader_election.cc:334] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.550s (SENT)
W20260927 19:38:19.987900 2111 leader_election.cc:334] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.608s (SENT)
I20260927 19:38:19.987895 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:19.845916 3595 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:19.989073 2111 leader_election.cc:334] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.670s (SENT)
I20260927 19:38:19.989549 2111 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:19.885543 3565 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:19.798331 3585 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:19.994578 3585 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.958676 3421 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.959508 3558 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:19.982410 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:19.974813 3463 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:19.881946 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:19.975195 3571 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.013576 1995 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:19.985117 3494 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.019191 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.019982 1995 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:20.018743 3494 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:19.999936 3480 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:20.018734 3463 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:20.021348 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:19.990906 3550 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:19.989766 1987 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:20.025250 1995 raft_consensus.cc:2362] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3. Re-sending same reply.
I20260927 19:38:20.013273 3559 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:20.025273 2111 leader_election.cc:334] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.720s (SENT)
I20260927 19:38:20.031172 2111 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:20.019016 3558 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:19.987850 3448 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.036592 2137 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:20.041510 3582 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.019623 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:20.042279 3582 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.024844 3463 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:20.027524 3545 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.054677 3598 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.055809 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:20.029170 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.036055 3480 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:20.045320 3559 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:20.071565 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 115) took 1044 ms. Trace:
I20260927 19:38:20.020953 3571 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.081621 2068 rpcz_store.cc:276] 0927 19:38:19.028261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.324412 (+296151us) service_pool.cc:227] Handling call
0927 19:38:19.363566 (+ 39154us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:19.363856 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:19.363904 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:19.377661 (+ 13757us) log.cc:842] Serialized 26 byte log entry
0927 19:38:19.737644 (+359983us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:20.070708 (+333064us) log.cc:842] Serialized 14 byte log entry
0927 19:38:20.070930 (+ 222us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.071037 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:20.071092 (+ 55us) raft_consensus.cc:1671] finished
0927 19:38:20.071105 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.071488 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":548927,"threads_started":2,"wal-append.queue_time_us":387130}
I20260927 19:38:20.072369 1991 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:20.089414 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.057204 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.067130 3545 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:20.033936 3550 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.091673 3559 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:20.063122 3589 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.049088 3558 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.058539 3463 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.128345 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:20.128611 3473 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.129189 3473 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.083250 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.114392 3372 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.143800 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:20.147287 2139 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 937) took 2427 ms (client timeout 1977 ms). Trace:
W20260927 19:38:20.147655 2139 rpcz_store.cc:269] 0927 19:38:17.717076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.717218 (+ 142us) service_pool.cc:227] Handling call
0927 19:38:17.896183 (+178965us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:20.143650 (+2247467us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02b7c 00007f7910a0273b 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:20.144625 (+ 975us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1130993280}
I20260927 19:38:20.131817 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.057236 3584 consensus_queue.cc:237] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.138590 3545 raft_consensus.cc:686] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:18.352005 3491 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.148353 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:20.122179 3595 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.156611 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.158653 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.160542 2654 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:20.161602 3413 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:20.162899 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:20.164616 2111 leader_election.cc:334] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.799s (SENT)
I20260927 19:38:20.153045 3473 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.165915 3473 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:20.166204 3564 consensus_meta.cc:287] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.788s user 0.009s sys 0.017s
I20260927 19:38:20.166539 3473 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:20.169518 2111 leader_election.cc:334] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.853s (SENT)
I20260927 19:38:20.170918 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.173043 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.179050 2654 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:20.180672 2111 leader_election.cc:334] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.868s (SENT)
I20260927 19:38:20.156211 2068 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:20.141821 2138 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:20.149612 3613 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.140837 3559 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.067022 3598 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.188066 3558 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.201290 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.202106 2137 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:20.203208 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:20.204540 3607 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
I20260927 19:38:20.147718 3597 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.183607 3413 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:20.166576 3564 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.218710 3550 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.192878 2683 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:20.224120 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.198079 2682 raft_consensus.cc:1226] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:20.199882 3558 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.233777 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1211) took 2441 ms. Trace:
I20260927 19:38:20.211009 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.236220 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:20.236110 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.238219 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:20.237035 2071 rpcz_store.cc:276] 0927 19:38:17.792057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.661245 (+869188us) service_pool.cc:227] Handling call
0927 19:38:18.661527 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:18.661794 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.661816 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:18.662200 (+ 384us) log.cc:842] Serialized 26 byte log entry
0927 19:38:18.748045 (+ 85845us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:18.753300 (+ 5255us) log.cc:842] Serialized 14 byte log entry
0927 19:38:18.803112 (+ 49812us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:18.803195 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:20.123611 (+1320416us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b1800336820. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:20.233412 (+109801us) raft_consensus.cc:1671] finished
0927 19:38:20.233534 (+ 122us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.233707 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2894,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":4143,"raft.queue_time_us":558961,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":32666,"spinlock_wait_cycles":7977600,"thread_start_us":111444,"threads_started":2,"wal-append.queue_time_us":103812}
I20260927 19:38:20.233281 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:20.241767 2071 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:20.243510 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1227) took 1002 ms. Trace:
I20260927 19:38:20.243813 2071 rpcz_store.cc:276] 0927 19:38:19.241220 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.241194 (+999974us) service_pool.cc:227] Handling call
0927 19:38:20.241471 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.243290 (+ 1819us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.243462 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:20.245159 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1228) took 1003 ms. Trace:
I20260927 19:38:20.245383 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:20.241438 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:20.246474 2686 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.246976 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:20.245462 2071 rpcz_store.cc:276] 0927 19:38:19.241517 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.244475 (+1002958us) service_pool.cc:227] Handling call
0927 19:38:20.244722 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.244872 (+ 150us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.244894 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:20.244931 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:20.244976 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.245027 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.245116 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:20.254719 2068 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:20.248610 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1229) took 1006 ms. Trace:
I20260927 19:38:20.256726 2071 rpcz_store.cc:276] 0927 19:38:19.241762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.247847 (+1006085us) service_pool.cc:227] Handling call
0927 19:38:20.248109 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.248252 (+ 143us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.248282 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:20.248362 (+ 80us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:20.248390 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.248438 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.248563 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:20.258448 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1230) took 1016 ms. Trace:
I20260927 19:38:20.258761 2071 rpcz_store.cc:276] 0927 19:38:19.242021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.257675 (+1015654us) service_pool.cc:227] Handling call
0927 19:38:20.257987 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.258146 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.258178 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:20.258214 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:20.258244 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.258291 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.258394 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:20.260334 3571 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.236155 3448 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.265784 3448 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:20.266149 3448 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:20.266512 3448 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.261691 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.256202 1987 raft_consensus.cc:2362] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3. Re-sending same reply.
W20260927 19:38:20.279155 1961 leader_election.cc:334] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.237s (SENT)
I20260927 19:38:20.279825 1961 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:20.208186 3613 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.261166 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.283054 2686 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:20.300805 2034 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:20.240337 3558 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.282547 3589 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:20.229501 3545 consensus_queue.cc:237] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.282682 3579 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.282707 3552 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.308706 3605 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:20.309556 3615 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.310285 3615 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.291215 3564 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:20.303771 3592 consensus_meta.cc:287] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.679s user 0.012s sys 0.000s
I20260927 19:38:20.316190 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.317194 2139 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:20.294049 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.288430 3582 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:20.288654 3617 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.324548 2068 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:20.327471 3598 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.312028 3601 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.305614 3473 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.324251 1987 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.320205 3491 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:20.336892 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.302065 3599 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.339063 3527 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260927 19:38:20.323258 3592 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:20.325794 3564 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:20.331027 3597 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:18.354840 3462 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.349246 3565 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:20.337447 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.344771 3473 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.346087 3589 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.334108 3601 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.361445 3599 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:20.315075 3552 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.343925 1991 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:20.332108 3579 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.370213 3559 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.380735 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.385710 3589 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.340641 3463 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.371843 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.332511 3617 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.389424 1987 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:20.336575 3494 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:20.420905 2066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 71) took 3450 ms (client timeout 1547 ms). Trace:
I20260927 19:38:20.381088 3615 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.387475 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.418484 3564 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:20.425611 3564 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.425618 2680 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:20.426862 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.427261 3552 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.427683 2681 raft_consensus.cc:2379] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:20.427600 3564 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:20.427866 3494 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:20.428866 3418 consensus_meta.cc:287] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.639s user 0.007s sys 0.007s
I20260927 19:38:20.383764 3473 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.386139 2068 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:20.388731 3589 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:20.421327 2066 rpcz_store.cc:269] 0927 19:38:16.969774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:16.970713 (+ 939us) service_pool.cc:227] Handling call
0927 19:38:20.419953 (+3449240us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b5400273dd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:20.420046 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":779360512}
I20260927 19:38:20.423763 3617 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.424794 2683 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:20.427798 2034 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:20.440514 3463 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.441104 3463 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:20.441803 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.442893 2681 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:20.429241 3418 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.430883 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.439355 3617 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.423678 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.447360 2139 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:20.449517 3494 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.450417 3494 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.451380 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.451529 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.431653 3615 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.452517 3615 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.453289 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:20.454106 2681 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:20.454509 3615 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.449517 3617 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.455264 1991 raft_consensus.cc:2354] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3.
W20260927 19:38:20.455860 2063 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1661) took 2928 ms (client timeout 1983 ms). Trace:
I20260927 19:38:20.456521 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.456652 3617 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.453197 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.461261 2142 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:20.459040 3418 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:20.462419 2142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 942) took 2128 ms (client timeout 2179 ms). Trace:
I20260927 19:38:20.457151 1991 raft_consensus.cc:2362] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2. Re-sending same reply.
W20260927 19:38:20.462868 2142 rpcz_store.cc:269] 0927 19:38:18.333602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.333743 (+ 141us) service_pool.cc:227] Handling call
0927 19:38:20.453027 (+2119284us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:20.462342 (+ 9315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116209024}
I20260927 19:38:20.429927 3558 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.464789 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:20.452548 2141 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 941) took 2122 ms (client timeout 2092 ms). Trace:
W20260927 19:38:20.466037 2141 rpcz_store.cc:269] 0927 19:38:18.330609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.331709 (+ 1100us) service_pool.cc:227] Handling call
0927 19:38:20.451276 (+2119567us) spinlock_profiling.cc:243] Waited 52.6 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:20.452472 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121093504}
I20260927 19:38:20.467070 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.467664 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:18.338426 3524 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:20.468803 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.470173 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.470019 3524 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.471319 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.472505 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.388319 3579 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.386772 3559 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.477737 2685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 272) took 1526 ms. Trace:
W20260927 19:38:20.456331 2063 rpcz_store.cc:269] 0927 19:38:17.527684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.688995 (+161311us) service_pool.cc:227] Handling call
0927 19:38:20.455643 (+2766648us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54002139d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:20.455775 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1049815808}
I20260927 19:38:20.463469 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.386788 3613 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.475951 3494 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.478130 2685 rpcz_store.cc:276] 0927 19:38:18.951848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.951981 (+ 133us) service_pool.cc:227] Handling call
0927 19:38:18.981926 (+ 29945us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:19.720029 (+738103us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.477510 (+757481us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b54004367d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:20.477684 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1676080384}
I20260927 19:38:20.426141 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.490420 3618 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.465263 3473 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.491668 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.492735 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 145) took 1599 ms. Trace:
W20260927 19:38:20.513585 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 46464.3us
I20260927 19:38:20.494680 3617 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:20.478842 3559 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.489034 3615 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.389169 3448 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.383030 3462 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.504066 3570 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.529824 3613 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:20.516767 3463 raft_consensus.cc:686] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:20.517201 3617 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.532197 1992 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:20.529942 3555 consensus_queue.cc:1046] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260927 19:38:20.513548 1989 rpcz_store.cc:276] 0927 19:38:18.893603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.902628 (+ 9025us) service_pool.cc:227] Handling call
0927 19:38:20.491793 (+1589165us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:20.492061 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.492280 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.492324 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:20.492380 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:20.492423 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.492497 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.492639 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7445888}
I20260927 19:38:20.541069 3555 raft_consensus.cc:3032] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:20.541374 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.542034 3555 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:20.543030 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.544242 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.544813 1989 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.551604 2141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:20.552392 2141 raft_consensus.cc:595] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:20.552843 2141 raft_consensus.cc:603] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:20.556927 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.522899 3553 consensus_queue.cc:1046] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260927 19:38:20.534564 3466 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.540169 3617 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:20.563614 2687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 276) took 1342 ms. Trace:
I20260927 19:38:20.525647 3615 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.565322 2687 rpcz_store.cc:276] 0927 19:38:19.222850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.240193 (+ 17343us) service_pool.cc:227] Handling call
0927 19:38:20.562955 (+1322762us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:20.563132 (+ 177us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.563285 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.563313 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:20.563354 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:20.563385 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.563437 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.563553 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12776320}
I20260927 19:38:20.531629 3613 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.539832 3570 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.597162 3473 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:20.598925 3473 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.566758 3466 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.601114 3615 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.603155 3473 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.586886 3463 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.604013 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46930 (request call id 134) took 1014 ms. Trace:
I20260927 19:38:20.607452 2066 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:20.614341 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.616781 1989 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:20.617966 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:20.618618 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 400667us
I20260927 19:38:20.619374 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.620121 1989 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:20.621012 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:20.621711 1992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 932) took 2280 ms (client timeout 2092 ms). Trace:
W20260927 19:38:20.622148 1992 rpcz_store.cc:269] 0927 19:38:18.343268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.400442 (+ 57174us) service_pool.cc:227] Handling call
0927 19:38:20.621536 (+2221094us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54003034d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:20.621646 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4747035392}
I20260927 19:38:20.621827 1989 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:20.627822 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.628815 1992 raft_consensus.cc:2362] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2. Re-sending same reply.
I20260927 19:38:20.629904 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.543162 3555 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.618721 3466 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.632869 3448 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.606495 3570 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.621270 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.625149 3462 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.645447 2066 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:20.637701 1992 raft_consensus.cc:2379] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:20.663618 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.664601 1992 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:20.647380 3559 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.522626 3618 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.612301 2138 rpcz_store.cc:276] 0927 19:38:19.590018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.591748 (+ 1730us) service_pool.cc:227] Handling call
0927 19:38:20.603823 (+1012075us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b5400122854. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:20.603953 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58379008}
I20260927 19:38:20.666016 1988 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.733598 1992 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:20.734951 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.735450 2654 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:20.736521 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.678067 3555 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:20.761220 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:20.761742 2111 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.129:39941 timed out after 1.980s (SENT) [suppressed 28 similar messages]
I20260927 19:38:20.764669 1992 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:20.769024 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.775993 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.776588 1992 raft_consensus.cc:2362] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2. Re-sending same reply.
I20260927 19:38:20.778044 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.778143 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.749989 3448 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.785404 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.786268 1992 raft_consensus.cc:2362] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3. Re-sending same reply.
I20260927 19:38:20.787271 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.763386 3613 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.763588 3416 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:20.797490 1992 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:20.798537 2111 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:20.819252 2141 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:20.820917 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.781924 3559 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.827447 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.797844 3466 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.830977 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.833500 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:20.829049 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.850683 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:20.852933 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.853691 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.854434 2143 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:20.855213 2143 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 938) took 3116 ms (client timeout 1835 ms). Trace:
I20260927 19:38:20.801504 3563 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260927 19:38:20.857673 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:20.823710 3473 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:20.855607 2143 rpcz_store.cc:269] 0927 19:38:17.738683 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.742819 (+ 4136us) service_pool.cc:227] Handling call
0927 19:38:18.236105 (+493286us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:20.853561 (+2617456us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02b7c 00007f7910a0273b 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:20.855155 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":329758208}
I20260927 19:38:20.824262 3576 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:20.822880 3613 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:20.829931 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.822786 3416 raft_consensus.cc:686] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:20.826508 3615 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:20.797871 3448 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.833773 3600 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:20.886602 3545 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:20.884473 3553 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:20.910195 1992 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:20.920370 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.901736 3620 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:20.832587 3372 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260927 19:38:20.882992 3628 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:20.831544 3559 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.921519 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.942646 1992 raft_consensus.cc:2362] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3. Re-sending same reply.
I20260927 19:38:20.944077 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:20.925463 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 67869.6us
I20260927 19:38:20.860285 3604 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:20.910619 3600 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:20.908296 3431 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:20.891551 3576 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.893162 3473 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.962589 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:20.952056 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:20.901844 3613 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.959604 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.968820 1992 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:20.962397 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.932035 3574 raft_consensus.cc:3032] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:20.972066 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.973346 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.854432 3466 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.939585 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:20.938265 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:20.974845 1992 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:20.974851 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:20.973642 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.948334 3553 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:20.919129 3615 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.921213 3545 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:20.993341 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.993722 3553 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:20.994226 1992 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:20.995692 3615 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.995549 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:20.972985 3524 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.500s user 0.040s sys 0.096s
I20260927 19:38:20.996800 1992 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:20.995776 3553 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.983582 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:20.997985 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.971606 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 947) took 1751 ms. Trace:
I20260927 19:38:21.002737 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:21.008219 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:21.013280 3633 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260927 19:38:21.014264 3473 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:20.974402 3574 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:21.015048 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 935) took 1293 ms. Trace:
I20260927 19:38:21.016347 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.967629 3527 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260927 19:38:21.021921 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:22.545994 2145 raft_consensus.cc:2396] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260927 19:38:22.546592 2145 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 969) took 1571 ms (client timeout 1814 ms). Trace:
W20260927 19:38:22.547042 2145 rpcz_store.cc:269] 0927 19:38:20.975676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:21.001635 (+ 25959us) service_pool.cc:227] Handling call
0927 19:38:22.546523 (+1544888us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.548290 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:22.551285 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08001130a0 after lost signal to thread 821
I20260927 19:38:20.965617 3604 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:22.552431 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08000face0 after lost signal to thread 1927
I20260927 19:38:22.553719 3604 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:22.554010 2092 debug-util.cc:398] Leaking SignalData structure 0x7b080039cdc0 after lost signal to thread 2020
W20260927 19:38:22.554338 2092 debug-util.cc:398] Leaking SignalData structure 0x7b080039cda0 after lost signal to thread 2021
I20260927 19:38:21.023077 3559 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:22.554970 3559 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:22.555531 3559 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:22.556613 3564 raft_consensus.cc:3032] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:22.557108 3564 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:22.557921 3559 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:21.001842 3594 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:22.557986 3564 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:21.033763 3613 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:21.034228 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:20.980405 3618 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:20.928566 3620 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.559705 1992 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
W20260927 19:38:22.560204 1992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1187) took 2525 ms (client timeout 2777 ms). Trace:
W20260927 19:38:22.560657 1992 rpcz_store.cc:269] 0927 19:38:20.034980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:21.033865 (+998885us) service_pool.cc:227] Handling call
0927 19:38:22.560143 (+1526278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.561221 3620 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:21.016268 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 1796) took 1239 ms. Trace:
I20260927 19:38:22.750474 2139 rpcz_store.cc:276] 0927 19:38:19.776742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.540462 (+763720us) service_pool.cc:227] Handling call
0927 19:38:20.561565 (+ 21103us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:20.590314 (+ 28749us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.590352 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:20.591012 (+ 660us) log.cc:842] Serialized 26 byte log entry
0927 19:38:20.746315 (+155303us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:20.955539 (+209224us) log.cc:842] Serialized 14 byte log entry
0927 19:38:20.955878 (+ 339us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.955966 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:21.016084 (+ 60118us) raft_consensus.cc:1671] finished
0927 19:38:21.016105 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:21.016217 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1050,"prepare.queue_time_us":268867,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":234,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":12179,"thread_start_us":291241,"threads_started":2,"wal-append.queue_time_us":229033,"wal-append.run_cpu_time_us":3067,"wal-append.run_wall_time_us":1189393}
W20260927 19:38:21.019963 3624 log.cc:924] Time spent T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
I20260927 19:38:21.021536 3524 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:22.754748 3524 raft_consensus.cc:686] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:22.756033 3524 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:22.757819 2686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 285) took 2393 ms. Trace:
I20260927 19:38:22.758361 2686 rpcz_store.cc:276] 0927 19:38:20.364580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.364730 (+ 150us) service_pool.cc:227] Handling call
0927 19:38:20.365096 (+ 366us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:20.365517 (+ 421us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.365547 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:20.369213 (+ 3666us) log.cc:842] Serialized 26 byte log entry
0927 19:38:20.654197 (+284984us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:20.655523 (+ 1326us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.655610 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:22.757454 (+2101844us) raft_consensus.cc:1671] finished
0927 19:38:22.757490 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:22.757710 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":152,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":2045,"thread_start_us":284674,"threads_started":1,"wal-append.queue_time_us":393650}
I20260927 19:38:21.021510 3547 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:21.049609 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 97656.5us
I20260927 19:38:21.054839 3576 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:21.059841 3635 log.cc:924] Time spent T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260927 19:38:21.063553 3617 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:21.015770 3574 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:21.088562 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1680) took 3405 ms. Trace:
I20260927 19:38:21.021510 2684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34936 (request call id 441) took 1352 ms. Trace:
I20260927 19:38:21.077005 2679 raft_consensus.cc:1226] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:21.087623 3600 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:21.110631 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 134107us
I20260927 19:38:18.361204 3368 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 2.714s
I20260927 19:38:20.933943 3416 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:21.055672 3448 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:21.032138 3466 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:21.252959 3621 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:21.437911 3615 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:21.487123 3558 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:21.488154 3557 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:21.679425 3555 consensus_meta.cc:287] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.899s user 0.006s sys 0.006s
W20260927 19:38:22.223246 3642 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.674s user 0.000s sys 0.000s
I20260927 19:38:22.240780 3648 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:22.274813 3649 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:22.378620 3650 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:22.380750 3627 log.cc:924] Time spent T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 1.551s user 0.000s sys 0.013s
I20260927 19:38:21.015889 2136 rpcz_store.cc:276] 0927 19:38:19.220355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.220926 (+ 571us) service_pool.cc:227] Handling call
0927 19:38:19.257259 (+ 36333us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:19.297315 (+ 40056us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:19.298363 (+ 1048us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:19.410821 (+112458us) log.cc:842] Serialized 26 byte log entry
0927 19:38:20.444095 (+1033274us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b180028a680. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e9411 00007f79139365cf
0927 19:38:20.879590 (+435495us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:20.886568 (+ 6978us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.886670 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:20.941561 (+ 54891us) raft_consensus.cc:1671] finished
0927 19:38:20.941582 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:20.968814 (+ 27232us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":177958,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":22416,"spinlock_wait_cycles":6121728,"thread_start_us":388449,"threads_started":1,"wal-append.queue_time_us":1511311,"wal-append.run_cpu_time_us":1690,"wal-append.run_wall_time_us":1051959}
I20260927 19:38:22.531358 3560 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:22.549145 2145 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 962) took 1585 ms (client timeout 1947 ms). Trace:
I20260927 19:38:21.032303 2062 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:20.998788 3550 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:22.559628 3564 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:20.995980 3545 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:21.016539 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54038.6us
I20260927 19:38:18.352221 3415 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:22.606176 2064 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1677) took 4924 ms (client timeout 2237 ms). Trace:
W20260927 19:38:22.614619 2652 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.623s user 0.019s sys 0.019s
I20260927 19:38:22.616081 3594 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:22.616156 3409 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:22.623926 2137 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 1805) took 2804 ms (client timeout 2190 ms). Trace:
W20260927 19:38:22.630862 2143 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 968) took 1656 ms (client timeout 1551 ms). Trace:
W20260927 19:38:22.640450 2065 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1232) took 2867 ms (client timeout 2792 ms). Trace:
I20260927 19:38:22.659392 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1226) took 3418 ms. Trace:
W20260927 19:38:22.660524 3634 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 1.710s user 0.016s sys 0.017s
I20260927 19:38:22.662938 2682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 433) took 4316 ms. Trace:
I20260927 19:38:20.980283 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:22.680310 3603 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:22.683089 3473 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:22.695354 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 133) took 3349 ms. Trace:
I20260927 19:38:22.696962 3494 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:22.700760 2141 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.743s user 0.034s sys 0.042s
I20260927 19:38:22.708791 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:22.715482 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:22.717291 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee2e0 after lost signal to thread 1892
W20260927 19:38:22.721819 3625 log.cc:924] Time spent T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 1.894s user 0.005s sys 0.011s
I20260927 19:38:22.722932 2061 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcc7a171b3e443d8f28fd70f8742470 with delete_type TABLET_DATA_TOMBSTONED (TS 4cc2df048f424e08803a390946004df0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48542
I20260927 19:38:22.724282 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1231) took 3482 ms. Trace:
I20260927 19:38:22.730191 3655 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:22.732649 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:22.737727 2070 raft_consensus.cc:1261] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:22.741585 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1237) took 2967 ms. Trace:
W20260927 19:38:22.742311 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.749s user 0.018s sys 0.018s
W20260927 19:38:22.745572 2653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.775s user 0.011s sys 0.019s
W20260927 19:38:22.746481 2092 debug-util.cc:398] Leaking SignalData structure 0x7b080039bd80 after lost signal to thread 2095
I20260927 19:38:22.862131 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1239) took 3087 ms. Trace:
I20260927 19:38:22.892861 3524 consensus_queue.cc:1046] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:22.893958 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1185) took 5734 ms. Trace:
I20260927 19:38:22.894129 3547 raft_consensus.cc:3032] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:22.894892 3547 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:22.895781 3547 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.896948 3547 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:22.897891 3653 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:22.898407 3653 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.898829 3617 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:22.766419 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1243) took 2990 ms. Trace:
I20260927 19:38:22.899729 2066 rpcz_store.cc:276] 0927 19:38:19.775590 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.666485 (+890895us) service_pool.cc:227] Handling call
0927 19:38:20.679829 (+ 13344us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:20.685603 (+ 5774us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.685648 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:20.686321 (+ 673us) log.cc:842] Serialized 26 byte log entry
0927 19:38:20.801233 (+114912us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:20.801268 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:20.801327 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:22.766171 (+1964844us) raft_consensus.cc:1671] finished
0927 19:38:22.766200 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:22.766360 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4991,"raft.queue_time_us":13508,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":38893,"thread_start_us":43103,"threads_started":1,"wal-append.queue_time_us":247161}
I20260927 19:38:22.766445 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:22.901494 2066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 122) took 1944 ms (client timeout 1513 ms). Trace:
I20260927 19:38:22.901610 2679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 270) took 3988 ms. Trace:
W20260927 19:38:22.901839 2066 rpcz_store.cc:269] 0927 19:38:20.957348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.901298 (+1943950us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:22.901418 (+ 120us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:22.902328 2679 rpcz_store.cc:276] 0927 19:38:18.912816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.921261 (+ 8445us) service_pool.cc:227] Handling call
0927 19:38:21.076449 (+2155188us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:21.076744 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:22.897762 (+1821018us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.901539 (+ 3777us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2814848}
I20260927 19:38:22.903785 3648 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.904218 3653 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:22.905507 3594 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.905956 3648 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:22.907714 3594 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:22.908208 2092 debug-util.cc:398] Leaking SignalData structure 0x7b080039bda0 after lost signal to thread 2096
I20260927 19:38:22.908612 3560 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:22.909337 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08003b9f60 after lost signal to thread 2169
W20260927 19:38:22.909703 2092 debug-util.cc:398] Leaking SignalData structure 0x7b08003b9f80 after lost signal to thread 2170
I20260927 19:38:22.909655 2067 rpcz_store.cc:276] 0927 19:38:19.240927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.458962 (+218035us) service_pool.cc:227] Handling call
0927 19:38:19.459365 (+ 403us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:19.671048 (+211683us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.659092 (+2988044us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540009f454. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:22.659327 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":479597824}
I20260927 19:38:22.910853 3560 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:22.911028 2092 debug-util.cc:398] Leaking SignalData structure 0x7b0800250f60 after lost signal to thread 2712
W20260927 19:38:22.910959 2067 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 125) took 1952 ms (client timeout 1768 ms). Trace:
I20260927 19:38:22.861743 3559 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:22.911554 2067 rpcz_store.cc:269] 0927 19:38:20.958017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.910792 (+1952775us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:22.910904 (+ 112us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:22.912175 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:22.901932 2139 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 963) took 1937 ms (client timeout 1987 ms). Trace:
I20260927 19:38:22.891955 2063 rpcz_store.cc:276] 0927 19:38:19.774967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.519455 (+744488us) service_pool.cc:227] Handling call
0927 19:38:20.519786 (+ 331us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:20.520172 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.520204 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:20.520733 (+ 529us) log.cc:842] Serialized 52 byte log entry
0927 19:38:22.750126 (+2229393us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b18000bf4a0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e9411 00007f79139365cf
0927 19:38:22.762565 (+ 12439us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:22.860975 (+ 98410us) log.cc:842] Serialized 14 byte log entry
0927 19:38:22.861568 (+ 593us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.861681 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:22.861738 (+ 57us) raft_consensus.cc:1671] finished
0927 19:38:22.861856 (+ 118us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:22.862053 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2835,"prepare.queue_time_us":97725,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":1101,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":6131,"spinlock_wait_cycles":4492672,"thread_start_us":110274,"threads_started":2,"wal-append.queue_time_us":2234276}
I20260927 19:38:22.912948 2136 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
W20260927 19:38:22.917346 2136 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 971) took 1939 ms (client timeout 2016 ms). Trace:
W20260927 19:38:22.917500 2138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 959) took 1956 ms (client timeout 2152 ms). Trace:
W20260927 19:38:22.917821 2136 rpcz_store.cc:269] 0927 19:38:20.977513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.911869 (+1934356us) service_pool.cc:227] Handling call
0927 19:38:22.917291 (+ 5422us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:22.917984 2138 rpcz_store.cc:269] 0927 19:38:20.961336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.965280 (+ 3944us) service_pool.cc:227] Handling call
0927 19:38:22.917440 (+1952160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.918802 3615 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.919602 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:22.920512 2138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 970) took 1943 ms (client timeout 2094 ms). Trace:
I20260927 19:38:22.920980 3615 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:22.914960 2092 debug-util.cc:398] Leaking SignalData structure 0x7b0800203c00 after lost signal to thread 2713
I20260927 19:38:22.939862 3603 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:22.929069 2682 rpcz_store.cc:276] 0927 19:38:18.346861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.392852 (+ 45991us) service_pool.cc:227] Handling call
0927 19:38:18.395509 (+ 2657us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.236264 (+1840755us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.662613 (+2426349us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54006ed054. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:22.662845 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4117126400}
W20260927 19:38:22.939677 2143 rpcz_store.cc:269] 0927 19:38:20.974923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:21.002379 (+ 27456us) service_pool.cc:227] Handling call
0927 19:38:22.630777 (+1628398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.942412 3603 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:22.942780 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:22.939870 2137 rpcz_store.cc:269] 0927 19:38:19.819398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.337148 (+517750us) service_pool.cc:227] Handling call
0927 19:38:22.623625 (+2286477us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54000d0cd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:22.623841 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1461996288}
I20260927 19:38:22.943483 2143 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
W20260927 19:38:22.944026 2143 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 974) took 1964 ms (client timeout 2150 ms). Trace:
I20260927 19:38:22.944550 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:22.922080 3600 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:22.912393 2067 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 131) took 1953 ms (client timeout 1817 ms). Trace:
I20260927 19:38:22.929883 2071 rpcz_store.cc:276] 0927 19:38:19.242303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.259605 (+1017302us) service_pool.cc:227] Handling call
0927 19:38:20.297080 (+ 37475us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.306372 (+ 9292us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:20.306414 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:20.306480 (+ 66us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:20.306522 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.724011 (+2417489us) spinlock_profiling.cc:243] Waited 9.07 ms on lock 0x7b5400214654. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:22.724067 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:22.724221 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20851456}
W20260927 19:38:22.907948 2062 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 119) took 3135 ms (client timeout 3042 ms). Trace:
I20260927 19:38:22.922415 3662 tablet_replica.cc:330] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: stopping tablet replica
W20260927 19:38:22.945693 2137 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 973) took 1966 ms (client timeout 2174 ms). Trace:
W20260927 19:38:22.903177 2066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 128) took 1944 ms (client timeout 1736 ms). Trace:
I20260927 19:38:22.922781 2144 raft_consensus.cc:2396] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260927 19:38:22.946373 2066 rpcz_store.cc:269] 0927 19:38:20.958472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.903021 (+1944549us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:22.903128 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:22.930457 2065 rpcz_store.cc:269] 0927 19:38:19.772734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.055539 (+282805us) service_pool.cc:227] Handling call
0927 19:38:22.640190 (+2584651us) spinlock_profiling.cc:243] Waited 2.57 s on lock 0x7b5400106154. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:22.640367 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5908849408}
I20260927 19:38:22.940163 3473 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:22.947177 2144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 939) took 5203 ms (client timeout 1755 ms). Trace:
I20260927 19:38:22.947481 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:22.947619 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:22.947507 2144 rpcz_store.cc:269] 0927 19:38:17.744070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.744220 (+ 150us) service_pool.cc:227] Handling call
0927 19:38:18.306834 (+562614us) spinlock_profiling.cc:243] Waited 4.45 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:22.708620 (+4401786us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02b7c 00007f7910a0273b 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:22.947096 (+238476us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186971776}
W20260927 19:38:22.913692 2139 rpcz_store.cc:269] 0927 19:38:20.964202 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.764319 (+1800117us) service_pool.cc:227] Handling call
0927 19:38:22.901880 (+137561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.948501 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 129) took 1989 ms. Trace:
I20260927 19:38:22.923873 3409 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:22.914062 2063 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1246) took 1959 ms (client timeout 1943 ms). Trace:
W20260927 19:38:22.924698 2064 rpcz_store.cc:269] 0927 19:38:17.682177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.765289 (+ 83112us) service_pool.cc:227] Handling call
0927 19:38:22.605893 (+4840604us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b5400322b54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:22.606057 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":964589440}
I20260927 19:38:22.924857 3542 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:22.925616 3558 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:22.926306 2684 rpcz_store.cc:276] 0927 19:38:19.668635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.673478 (+ 4843us) service_pool.cc:227] Handling call
0927 19:38:20.938092 (+1264614us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:21.021443 (+ 83351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4313856}
I20260927 19:38:22.927574 3576 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:22.921558 3494 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:22.928846 3574 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:22.929132 3466 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:22.929760 3560 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:22.915159 3650 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:22.928965 3448 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:22.911579 2653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.941s user 0.014s sys 0.022s
W20260927 19:38:22.916255 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.923s user 0.021s sys 0.021s
I20260927 19:38:22.931659 3617 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:22.931228 3621 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:22.916209 2110 connection.cc:232] RPC call timeout handler was delayed by -1.168737426000007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
I20260927 19:38:22.916494 3649 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:22.929288 2138 rpcz_store.cc:269] 0927 19:38:20.977217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.919303 (+1942086us) service_pool.cc:227] Handling call
0927 19:38:22.920457 (+ 1154us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:22.916633 2652 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.940s user 0.020s sys 0.024s
W20260927 19:38:22.913551 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee740 after lost signal to thread 1927
W20260927 19:38:22.916308 2145 rpcz_store.cc:269] 0927 19:38:20.963672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.547954 (+1584282us) service_pool.cc:227] Handling call
0927 19:38:22.549091 (+ 1137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.912006 3559 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:22.929777 2142 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:22.948974 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:22.561710 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:22.651103 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:22.722692 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 238) took 8068 ms. Trace:
W20260927 19:38:22.313879 1960 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.834s user 0.014s sys 0.004s
W20260927 19:38:21.487689 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 294646us
W20260927 19:38:21.075809 1987 consensus_meta.cc:287] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.001s
I20260927 19:38:21.098287 1988 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:21.057695 1989 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:21.072479 1991 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.553s user 0.011s sys 0.002s
W20260927 19:38:22.275513 1961 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.000s sys 0.009s
I20260927 19:38:21.015447 1993 rpcz_store.cc:276] 0927 19:38:19.721211 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.721922 (+ 711us) service_pool.cc:227] Handling call
0927 19:38:20.939483 (+1217561us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02bce 000000000034aea9 00007f791882ea30 00007f79188f47b4 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:21.014975 (+ 75492us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36731264}
I20260927 19:38:22.725219 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:22.733891 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1179) took 2748 ms (client timeout 1901 ms). Trace:
I20260927 19:38:22.927815 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:22.948207 2066 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:22.948854 2065 rpcz_store.cc:276] 0927 19:38:20.958624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.947369 (+1988745us) service_pool.cc:227] Handling call
0927 19:38:22.948448 (+ 1079us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.916359 2068 rpcz_store.cc:276] 0927 19:38:19.774602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.440706 (+666104us) service_pool.cc:227] Handling call
0927 19:38:20.440960 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:22.377501 (+1936541us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:22.377547 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:22.377605 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:22.377648 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.741280 (+363632us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400323f54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:22.741315 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:22.741520 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4452711168}
I20260927 19:38:23.139997 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 130) took 2181 ms. Trace:
I20260927 19:38:22.894621 2070 rpcz_store.cc:276] 0927 19:38:17.159722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.300491 (+1140769us) service_pool.cc:227] Handling call
0927 19:38:22.737296 (+4436805us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:22.891588 (+154292us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.893730 (+ 2142us) spinlock_profiling.cc:243] Waited 4.44 s on lock 0x7b54002139d0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:22.893875 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10202708352}
W20260927 19:38:23.140658 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 951) took 2166 ms (client timeout 1909 ms). Trace:
I20260927 19:38:22.928313 2069 rpcz_store.cc:276] 0927 19:38:17.683104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.595975 (+912871us) service_pool.cc:227] Handling call
0927 19:38:18.601539 (+ 5564us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:18.614359 (+ 12820us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:18.614419 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:18.614478 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:18.614521 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:21.088205 (+2473684us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b5400323554. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:21.088275 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:21.088449 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8676096}
W20260927 19:38:23.141403 1993 rpcz_store.cc:269] 0927 19:38:20.974208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.140025 (+2165817us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.140350 (+ 325us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:23.142918 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56618 (request call id 158) took 2894 ms (client timeout 2658 ms). Trace:
I20260927 19:38:23.143282 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1248) took 2186 ms. Trace:
I20260927 19:38:23.143525 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 121) took 2186 ms. Trace:
W20260927 19:38:23.143790 2063 rpcz_store.cc:269] 0927 19:38:20.955061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.913929 (+1958868us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:22.914015 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:22.945715 2067 rpcz_store.cc:269] 0927 19:38:20.958916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.912248 (+1953332us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:22.912347 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.140689 2066 rpcz_store.cc:276] 0927 19:38:20.958776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.947226 (+1988450us) service_pool.cc:227] Handling call
0927 19:38:23.139712 (+192486us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.146925 1992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1233) took 2295 ms (client timeout 1980 ms). Trace:
I20260927 19:38:23.142957 2065 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:23.143947 2069 rpcz_store.cc:276] 0927 19:38:20.957110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.142116 (+2185006us) service_pool.cc:227] Handling call
0927 19:38:23.142720 (+ 604us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.143087 (+ 367us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.143107 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.143161 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.143213 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.143267 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.143390 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.147497 1992 rpcz_store.cc:269] 0927 19:38:20.851630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.561448 (+1709818us) service_pool.cc:227] Handling call
0927 19:38:23.146636 (+585188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.148110 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1245) took 3356 ms. Trace:
I20260927 19:38:23.148610 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 132) took 2189 ms. Trace:
I20260927 19:38:23.148689 2065 rpcz_store.cc:276] 0927 19:38:19.791936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.140560 (+3348624us) service_pool.cc:227] Handling call
0927 19:38:23.141737 (+ 1177us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.147774 (+ 6037us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.148025 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39680}
I20260927 19:38:22.946208 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.151296 2071 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
W20260927 19:38:23.152033 2071 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 127) took 2193 ms (client timeout 2658 ms). Trace:
W20260927 19:38:23.152504 2071 rpcz_store.cc:269] 0927 19:38:20.958332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.945936 (+1987604us) service_pool.cc:227] Handling call
0927 19:38:23.151949 (+206013us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:22.945842 2062 rpcz_store.cc:269] 0927 19:38:19.772205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.121779 (+349574us) service_pool.cc:227] Handling call
0927 19:38:22.907738 (+2785959us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b54000fd554. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:22.907887 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2076517632}
W20260927 19:38:23.153923 1960 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.674s user 0.014s sys 0.008s
I20260927 19:38:23.149315 2066 rpcz_store.cc:276] 0927 19:38:20.959063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.147002 (+2187939us) service_pool.cc:227] Handling call
0927 19:38:23.147507 (+ 505us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.147837 (+ 330us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.147860 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.147915 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.148323 (+ 408us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.148391 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.148542 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.158012 1989 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56618 (request call id 154) took 3109 ms (client timeout 2223 ms). Trace:
W20260927 19:38:23.158593 1989 rpcz_store.cc:269] 0927 19:38:20.048830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.620765 (+571935us) service_pool.cc:227] Handling call
0927 19:38:23.157760 (+2536995us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.159590 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 231694us
W20260927 19:38:23.159966 1961 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.523s user 0.003s sys 0.012s
W20260927 19:38:23.160501 1988 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56618 (request call id 144) took 4267 ms (client timeout 2106 ms). Trace:
I20260927 19:38:23.143795 2070 rpcz_store.cc:276] 0927 19:38:20.956846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.141556 (+2184710us) service_pool.cc:227] Handling call
0927 19:38:23.142192 (+ 636us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.142682 (+ 490us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.142712 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.142770 (+ 58us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:23.142815 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.142976 (+ 161us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.143114 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.948894 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:23.167644 2141 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.210s user 0.036s sys 0.050s
I20260927 19:38:23.167893 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:23.168537 2144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 972) took 2190 ms (client timeout 2474 ms). Trace:
W20260927 19:38:23.169023 2144 rpcz_store.cc:269] 0927 19:38:20.977780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.948665 (+1970885us) service_pool.cc:227] Handling call
0927 19:38:23.168357 (+219692us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.169394 2142 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 961) took 2206 ms (client timeout 1977 ms). Trace:
W20260927 19:38:23.169811 2142 rpcz_store.cc:269] 0927 19:38:20.963120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.715136 (+1752016us) service_pool.cc:227] Handling call
0927 19:38:23.169172 (+454036us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.170222 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:23.171025 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:23.171833 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.172545 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.173127 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 153) took 2350 ms. Trace:
I20260927 19:38:23.150098 1990 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:23.173321 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:23.175160 1990 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1192) took 3059 ms (client timeout 1670 ms). Trace:
W20260927 19:38:23.163585 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 2.18843e+06us
W20260927 19:38:22.946125 2137 rpcz_store.cc:269] 0927 19:38:20.979170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.944269 (+1965099us) service_pool.cc:227] Handling call
0927 19:38:22.945584 (+ 1315us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:22.922957 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.229285 3545 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.167164 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:23.176563 3432 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:23.177726 3334 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.177277 3557 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.178676 3555 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:23.179116 3594 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:23.180368 3615 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:23.180848 3584 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.184533 3409 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:23.145660 1996 rpcz_store.cc:269] 0927 19:38:19.985496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.733706 (+2748210us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:22.733812 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.144639 1995 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:23.149220 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1247) took 2193 ms. Trace:
W20260927 19:38:23.148957 1992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1209) took 2961 ms (client timeout 2792 ms). Trace:
I20260927 19:38:23.119429 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 141) took 4230 ms. Trace:
I20260927 19:38:23.058785 3564 raft_consensus.cc:465] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Not starting pre-election -- already a leader
W20260927 19:38:23.143613 1993 rpcz_store.cc:269] 0927 19:38:20.248226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.142617 (+2894391us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.142791 (+ 174us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.200434 3560 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:23.201102 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 220719us
I20260927 19:38:22.949725 3473 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.202298 3576 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.203749 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 233) took 4893 ms. Trace:
I20260927 19:38:23.204648 3649 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.208041 3559 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.215642 3494 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.216025 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:23.258773 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 955) took 2280 ms (client timeout 2094 ms). Trace:
W20260927 19:38:23.259194 1996 rpcz_store.cc:269] 0927 19:38:20.978789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.258582 (+2279793us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.258716 (+ 134us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.259644 3584 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:23.260257 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 946) took 2299 ms (client timeout 2152 ms). Trace:
W20260927 19:38:23.260583 1996 rpcz_store.cc:269] 0927 19:38:20.960714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.260034 (+2299320us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.260188 (+ 154us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.221273 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.223673 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:22.932569 2140 rpcz_store.cc:276] 0927 19:38:19.346364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:19.346528 (+ 164us) service_pool.cc:227] Handling call
0927 19:38:19.346910 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:19.585039 (+238129us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:22.695000 (+3109961us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400264154. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:22.695262 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":537670912}
W20260927 19:38:23.262744 1995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1182) took 3245 ms (client timeout 1996 ms). Trace:
W20260927 19:38:23.263264 1995 rpcz_store.cc:269] 0927 19:38:20.017837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.028214 (+ 10377us) service_pool.cc:227] Handling call
0927 19:38:22.725046 (+2696832us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:23.262657 (+537611us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2662272}
I20260927 19:38:23.264010 3594 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:23.264456 1995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 950) took 2294 ms (client timeout 2160 ms). Trace:
W20260927 19:38:23.264858 1995 rpcz_store.cc:269] 0927 19:38:20.970525 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.264192 (+2293667us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.264379 (+ 187us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:23.238317 2136 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 966) took 2269 ms (client timeout 2160 ms). Trace:
I20260927 19:38:23.265237 3334 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:23.265549 2136 rpcz_store.cc:269] 0927 19:38:20.968903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.922701 (+1953798us) service_pool.cc:227] Handling call
0927 19:38:23.238123 (+315422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.230865 3648 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.267262 3557 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.267668 3334 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.267122 3648 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:23.234185 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee800 after lost signal to thread 1942
I20260927 19:38:22.915023 3655 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:23.270423 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002eeca0 after lost signal to thread 1987
W20260927 19:38:23.270988 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002eed60 after lost signal to thread 1993
W20260927 19:38:23.271322 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002eed80 after lost signal to thread 1994
I20260927 19:38:23.271790 3655 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:23.272207 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002fd000 after lost signal to thread 2020
W20260927 19:38:23.272588 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002d3740 after lost signal to thread 2021
I20260927 19:38:23.272804 3576 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:23.273003 2709 debug-util.cc:398] Leaking SignalData structure 0x7b0800398d80 after lost signal to thread 2034
I20260927 19:38:23.229596 2144 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:23.235018 3558 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.203814 3617 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.273586 1893 rpcz_store.cc:276] 0927 19:38:14.654343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:14.655149 (+ 806us) service_pool.cc:227] Handling call
0927 19:38:22.722590 (+8067441us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:14.980950 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:15.062435 (+ 81485us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:15.070701 (+ 8266us) write_op.cc:435] Acquired schema lock
0927 19:38:15.074112 (+ 3411us) tablet.cc:660] Decoding operations
0927 19:38:15.159008 (+ 84896us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:15.159065 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:15.161927 (+ 2862us) tablet.cc:683] Acquiring locks for 4 operations
0927 19:38:15.356639 (+194712us) tablet.cc:699] Row locks acquired
0927 19:38:15.360953 (+ 4314us) write_op.cc:260] PREPARE: finished
0927 19:38:15.373637 (+ 12684us) write_op.cc:270] Start()
0927 19:38:15.388453 (+ 14816us) write_op.cc:276] Timestamp: P: 1790537895370948 usec, L: 0
0927 19:38:15.388480 (+ 27us) op_driver.cc:331] REPLICATION: starting
0927 19:38:15.497464 (+108984us) log.cc:842] Serialized 1691 byte log entry
0927 19:38:16.565118 (+1067654us) op_driver.cc:444] REPLICATION: finished
0927 19:38:16.881011 (+315893us) write_op.cc:301] APPLY: starting
0927 19:38:16.906907 (+ 25896us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:16.939967 (+ 33060us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:16.952286 (+ 12319us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:16.953006 (+ 720us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:16.953247 (+ 241us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0927 19:38:16.953315 (+ 68us) write_op.cc:312] APPLY: finished
0927 19:38:17.013135 (+ 59820us) log.cc:842] Serialized 41 byte log entry
0927 19:38:17.013323 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:17.013541 (+ 218us) write_op.cc:454] Released schema lock
0927 19:38:17.013580 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:17.013832 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:17.013862 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3925,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":264884,"apply.run_cpu_time_us":3613,"apply.run_wall_time_us":142143,"num_ops":4,"prepare.queue_time_us":1027,"prepare.run_cpu_time_us":6105,"prepare.run_wall_time_us":566699,"replication_time_us":1159774,"thread_start_us":182431,"threads_started":1}]]}
I20260927 19:38:23.281059 3662 raft_consensus.cc:2229] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:23.228189 3466 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:23.233559 1989 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1213) took 3044 ms (client timeout 2813 ms). Trace:
I20260927 19:38:23.173744 2141 rpcz_store.cc:276] 0927 19:38:20.823048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.880164 (+ 57116us) service_pool.cc:227] Handling call
0927 19:38:20.880600 (+ 436us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:20.884252 (+ 3652us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 19:38:20.957096 (+ 72844us) log.cc:842] Serialized 14 byte log entry
0927 19:38:23.172356 (+2215260us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.172479 (+ 123us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:23.172613 (+ 134us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.172748 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.173027 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3660,"prepare.queue_time_us":131167,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":16948,"thread_start_us":2265039,"threads_started":2,"wal-append.queue_time_us":380817}
W20260927 19:38:23.237075 1961 connection.cc:232] RPC call timeout handler was delayed by -1.4583145589998594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20260927 19:38:23.232724 1892 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 19:38:22.944361 2143 rpcz_store.cc:269] 0927 19:38:20.979444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.942510 (+1963066us) service_pool.cc:227] Handling call
0927 19:38:22.943969 (+ 1459us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.248210 1992 rpcz_store.cc:269] 0927 19:38:20.187794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.148729 (+2960935us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.148899 (+ 170us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.180979 3368 consensus_queue.cc:1046] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260927 19:38:23.247997 2068 rpcz_store.cc:276] 0927 19:38:20.956089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.140682 (+2184593us) service_pool.cc:227] Handling call
0927 19:38:23.148323 (+ 7641us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.148771 (+ 448us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.148793 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.148844 (+ 51us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:23.148911 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.148956 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.149047 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68352}
W20260927 19:38:23.161141 1988 rpcz_store.cc:269] 0927 19:38:18.893248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.153255 (+1260007us) service_pool.cc:227] Handling call
0927 19:38:20.621128 (+467873us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:23.160265 (+2539137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17632000}
I20260927 19:38:23.248222 1994 rpcz_store.cc:276] 0927 19:38:18.889217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:18.889370 (+ 153us) service_pool.cc:227] Handling call
0927 19:38:18.941019 (+ 51649us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.102474 (+4161455us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.102590 (+ 116us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.102677 (+ 87us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.102759 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.118700 (+ 15941us) spinlock_profiling.cc:243] Waited 4.16 s on lock 0x7b540036f954. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:23.118803 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.119146 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9568296832}
I20260927 19:38:23.251571 1890 rpcz_store.cc:276] 0927 19:38:18.310663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.463587 (+2152924us) service_pool.cc:227] Handling call
0927 19:38:23.203451 (+2739864us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2183697,"reported_tablets":1}
W20260927 19:38:23.257899 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 957) took 2266 ms (client timeout 2016 ms). Trace:
I20260927 19:38:23.257988 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.274495 3648 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.262266 3584 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.188694 3448 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.270354 3650 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:23.277339 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1235) took 2350 ms (client timeout 2191 ms). Trace:
W20260927 19:38:23.190668 1990 rpcz_store.cc:269] 0927 19:38:20.116161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.206280 (+ 90119us) service_pool.cc:227] Handling call
0927 19:38:22.650896 (+2444616us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:23.175029 (+524133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4877696}
I20260927 19:38:23.257128 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:23.218370 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 57365.5us
W20260927 19:38:23.301124 1996 rpcz_store.cc:269] 0927 19:38:20.925124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.261275 (+2336151us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.275519 (+ 14244us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:23.290028 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08002339a0 after lost signal to thread 2037
W20260927 19:38:23.307437 1995 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 956) took 2327 ms (client timeout 2174 ms). Trace:
W20260927 19:38:23.309883 1995 rpcz_store.cc:269] 0927 19:38:20.980302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.304412 (+2324110us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.307372 (+ 2960us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:23.308204 1992 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 949) took 2340 ms (client timeout 2246 ms). Trace:
W20260927 19:38:23.312296 1992 rpcz_store.cc:269] 0927 19:38:20.968068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.304484 (+2336416us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.308156 (+ 3672us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:23.288090 1989 rpcz_store.cc:269] 0927 19:38:20.188902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.233303 (+3044401us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.233482 (+ 179us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260927 19:38:23.297016 1993 rpcz_store.cc:269] 0927 19:38:20.991863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.257628 (+2265765us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:23.257818 (+ 190us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:23.318391 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 124) took 2360 ms. Trace:
I20260927 19:38:23.318737 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1180) took 3332 ms. Trace:
I20260927 19:38:23.319249 1993 rpcz_store.cc:276] 0927 19:38:19.986537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.317293 (+3330756us) service_pool.cc:227] Handling call
0927 19:38:23.317834 (+ 541us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.318180 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.318209 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.318266 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:23.318297 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.318374 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.318637 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.319041 2063 rpcz_store.cc:276] 0927 19:38:20.957866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.145099 (+2187233us) service_pool.cc:227] Handling call
0927 19:38:23.145669 (+ 570us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.317112 (+171443us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.317160 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.317213 (+ 53us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.317251 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.318061 (+ 810us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54000fe454. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:23.318125 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.318296 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378268544}
I20260927 19:38:23.314091 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:23.322749 1994 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:23.326517 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 123) took 2289 ms. Trace:
I20260927 19:38:23.325212 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.325166 2685 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:23.336681 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 2.31147e+06us
I20260927 19:38:23.313180 1987 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:23.338308 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.324146 1991 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:23.328971 3650 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.327929 2064 rpcz_store.cc:276] 0927 19:38:20.957701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.144400 (+2186699us) service_pool.cc:227] Handling call
0927 19:38:23.145011 (+ 611us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.145639 (+ 628us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 19:38:23.146458 (+ 819us) log.cc:842] Serialized 14 byte log entry
0927 19:38:23.245038 (+ 98580us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.245109 (+ 71us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:23.245164 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.245245 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.246890 (+ 1645us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40617,"prepare.queue_time_us":42487,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":49633,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":11258,"thread_start_us":49678,"threads_started":1,"wal-append.queue_time_us":164295}
I20260927 19:38:23.350831 2037 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.349097 1989 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:23.353681 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.354346 3466 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.354413 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1198) took 3236 ms. Trace:
I20260927 19:38:23.354647 2684 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:23.355093 1989 rpcz_store.cc:276] 0927 19:38:20.117833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.335286 (+3217453us) service_pool.cc:227] Handling call
0927 19:38:23.348531 (+ 13245us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.354075 (+ 5544us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.354326 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.342694 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:23.344826 3558 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:23.364547 1961 connection.cc:232] RPC call timeout handler was delayed by -1.3665145589999383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260927 19:38:23.365079 1961 connection.cc:232] RPC call timeout handler was delayed by -1.3629145589999325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
W20260927 19:38:23.365506 1961 connection.cc:232] RPC call timeout handler was delayed by -1.2666145589998905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
W20260927 19:38:23.365922 1961 connection.cc:232] RPC call timeout handler was delayed by -1.2387145589999591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260927 19:38:23.353189 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.366786 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.367419 2685 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:23.366655 1961 connection.cc:232] RPC call timeout handler was delayed by -1.2072145589999081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
I20260927 19:38:23.345995 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:23.363878 3368 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.373443 3368 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.376479 3368 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.377604 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.377808 3448 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.370944 3650 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.379017 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.380229 3560 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:23.380537 2653 connection.cc:232] RPC call timeout handler was delayed by -2.1901886780000268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260927 19:38:23.380980 2653 connection.cc:232] RPC call timeout handler was delayed by -1.5269452210001191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
I20260927 19:38:23.381187 3650 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.342537 3648 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:23.382252 3648 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.382781 3584 raft_consensus.cc:3032] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:23.383200 3584 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:23.369134 1991 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1197) took 3251 ms (client timeout 1853 ms). Trace:
I20260927 19:38:23.384572 3648 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.376148 3655 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260927 19:38:23.384943 1991 rpcz_store.cc:269] 0927 19:38:20.117574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.491382 (+373808us) service_pool.cc:227] Handling call
0927 19:38:23.362803 (+2871421us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.385847 2141 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:23.384502 3416 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:23.387276 3416 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.388603 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.389474 2145 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:23.389698 3416 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.389832 1959 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:23.391611 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:23.368037 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:23.392118 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.392343 2144 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:23.384485 3584 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.392539 1990 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:23.392804 2686 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:23.353726 3576 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:23.393796 3584 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:23.394667 3542 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:23.395195 3542 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:23.395546 3542 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:23.391726 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.396191 1989 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:23.396900 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1202) took 3277 ms. Trace:
I20260927 19:38:23.362623 3617 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.342902 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:23.326942 3545 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:23.362280 1987 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1199) took 3244 ms (client timeout 1868 ms). Trace:
W20260927 19:38:23.397732 2653 connection.cc:232] RPC call timeout handler was delayed by -1.5010157819999677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20260927 19:38:23.362905 1993 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:23.402940 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.409475 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.407327 1996 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:23.403311 2653 connection.cc:232] RPC call timeout handler was delayed by -1.2916051460000517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1817s for a response
I20260927 19:38:23.339382 1892 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:23.406044 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.404036 1995 raft_consensus.cc:595] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:23.411458 3662 pending_rounds.cc:62] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Trying to abort 1 pending ops.
W20260927 19:38:23.411113 2653 connection.cc:232] RPC call timeout handler was delayed by -1.0349958320000496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3042s for a response
W20260927 19:38:23.403010 1987 rpcz_store.cc:269] 0927 19:38:20.118090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:20.547702 (+429612us) service_pool.cc:227] Handling call
0927 19:38:23.357653 (+2809951us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.412765 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:23.413897 1962 connection.cc:232] RPC call timeout handler was delayed by -2.0604121809999469s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
I20260927 19:38:23.413774 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.405978 3466 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 19:38:23.414383 1962 connection.cc:232] RPC call timeout handler was delayed by -2.0325121809999018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260927 19:38:23.415036 1994 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:23.421545 1892 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:23.422570 2144 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:23.423833 2688 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:23.423506 1892 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 19:38:23.425869 2709 debug-util.cc:398] Leaking SignalData structure 0x7b0800372ea0 after lost signal to thread 2095
W20260927 19:38:23.426316 2709 debug-util.cc:398] Leaking SignalData structure 0x7b0800370e60 after lost signal to thread 2096
W20260927 19:38:23.428028 2709 debug-util.cc:398] Leaking SignalData structure 0x7b0800396340 after lost signal to thread 2169
W20260927 19:38:23.428416 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08003963a0 after lost signal to thread 2170
I20260927 19:38:23.430022 3594 consensus_queue.cc:1046] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:23.431346 3594 raft_consensus.cc:3032] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:23.431825 3594 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:23.432667 2108 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:23.412493 3621 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.415750 3466 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:23.433101 3594 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.434161 3466 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:23.434965 3594 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:23.435729 1992 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:23.436398 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1177) took 3551 ms. Trace:
I20260927 19:38:23.436792 1992 rpcz_store.cc:276] 0927 19:38:19.885319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.333652 (+3448333us) service_pool.cc:227] Handling call
0927 19:38:23.435342 (+101690us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02bce 000000000034aea9 00007f791882ea30 00007f79188f47b4 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:23.435504 (+ 162us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.436132 (+ 628us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.436315 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2695552}
I20260927 19:38:23.438236 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34128 (request call id 16) took 2857 ms. Trace:
I20260927 19:38:23.410815 1991 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:23.397301 1989 rpcz_store.cc:276] 0927 19:38:20.119349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.356157 (+3236808us) service_pool.cc:227] Handling call
0927 19:38:23.357207 (+ 1050us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.396600 (+ 39393us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.396826 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.449997 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1203) took 3330 ms. Trace:
I20260927 19:38:23.450359 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:23.451077 3650 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.451367 1989 raft_consensus.cc:2362] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3. Re-sending same reply.
I20260927 19:38:23.451710 3650 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.438588 1995 rpcz_store.cc:276] 0927 19:38:20.580629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.326572 (+2745943us) service_pool.cc:227] Handling call
0927 19:38:23.437998 (+111426us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.438895 1994 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1212) took 3227 ms (client timeout 3224 ms). Trace:
W20260927 19:38:23.438601 1988 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1200) took 3298 ms (client timeout 3308 ms). Trace:
W20260927 19:38:23.441257 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 88308.5us
I20260927 19:38:23.445796 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:23.447069 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:23.458928 1994 rpcz_store.cc:269] 0927 19:38:20.188633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.305567 (+3116934us) service_pool.cc:227] Handling call
0927 19:38:23.416336 (+110769us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.452716 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:23.460311 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:23.460387 1988 rpcz_store.cc:269] 0927 19:38:20.118370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.322684 (+3204314us) service_pool.cc:227] Handling call
0927 19:38:23.416256 (+ 93572us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.416112 3662 pending_rounds.cc:68] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7334043190765961216 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } attrs { replace: true } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } } }
I20260927 19:38:23.464903 3567 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.450434 1991 rpcz_store.cc:276] 0927 19:38:20.119609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.385877 (+3266268us) service_pool.cc:227] Handling call
0927 19:38:23.386317 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.449690 (+ 63373us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.449933 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.438766 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1184) took 3398 ms. Trace:
W20260927 19:38:23.481899 2111 connection.cc:232] RPC call timeout handler was delayed by -1.6288062389999141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3308s for a response
I20260927 19:38:23.452991 3545 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.468612 3494 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election won for term 3
W20260927 19:38:23.495673 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 63270us
W20260927 19:38:23.493906 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 78329.2us
I20260927 19:38:23.497052 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 126) took 2444 ms. Trace:
I20260927 19:38:23.498131 1996 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:23.479350 3662 raft_consensus.cc:2894] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
W20260927 19:38:23.500726 1996 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 954) took 2520 ms (client timeout 2474 ms). Trace:
W20260927 19:38:23.501554 1996 rpcz_store.cc:269] 0927 19:38:20.978451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.321510 (+2343059us) service_pool.cc:227] Handling call
0927 19:38:23.498629 (+177119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.500990 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.481145 1989 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:23.506104 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1208) took 3318 ms. Trace:
I20260927 19:38:23.495265 3558 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:23.507804 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.508849 1996 rpcz_store.cc:276] 0927 19:38:20.187456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.502503 (+3315047us) service_pool.cc:227] Handling call
0927 19:38:23.502997 (+ 494us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.505588 (+ 2591us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.505633 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.505687 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:23.505734 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.505834 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.506006 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.507140 1992 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260927 19:38:23.516274 2111 connection.cc:232] RPC call timeout handler was delayed by -1.1323456740000211s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2792s for a response
W20260927 19:38:23.516873 2111 connection.cc:232] RPC call timeout handler was delayed by -1.1134456739999905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28130000000000005s for a response
I20260927 19:38:23.517805 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 156) took 3362 ms. Trace:
I20260927 19:38:23.518301 1992 rpcz_store.cc:276] 0927 19:38:20.155720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.498535 (+3342815us) service_pool.cc:227] Handling call
0927 19:38:23.506608 (+ 8073us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.514966 (+ 8358us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.517720 (+ 2754us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.519908 1962 connection.cc:232] RPC call timeout handler was delayed by -1.9571766579999803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20260927 19:38:23.498749 2067 rpcz_store.cc:276] 0927 19:38:20.958185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.145790 (+2187605us) service_pool.cc:227] Handling call
0927 19:38:23.146325 (+ 535us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.146792 (+ 467us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 19:38:23.245931 (+ 99139us) log.cc:842] Serialized 14 byte log entry
0927 19:38:23.371763 (+125832us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.371821 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.371862 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.371987 (+ 125us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.402203 (+ 30216us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":41869,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":35109,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":1086,"thread_start_us":174572,"threads_started":2,"wal-append.queue_time_us":136895}
I20260927 19:38:23.526253 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.480592 1993 rpcz_store.cc:276] 0927 19:38:20.019782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.320794 (+3301012us) service_pool.cc:227] Handling call
0927 19:38:23.325905 (+ 5111us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.408663 (+ 82758us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.416363 (+ 7700us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:23.527482 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 74176.2us
I20260927 19:38:23.520896 2108 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:23.502086 3494 raft_consensus.cc:686] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:23.530454 1994 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:23.531335 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1204) took 3411 ms. Trace:
I20260927 19:38:23.528002 3432 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:23.531766 1994 rpcz_store.cc:276] 0927 19:38:20.119853 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.483029 (+3363176us) service_pool.cc:227] Handling call
0927 19:38:23.508457 (+ 25428us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.531101 (+ 22644us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.531289 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.532233 3432 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.534444 3558 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:23.534847 3432 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.537103 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.546115 2683 raft_consensus.cc:2379] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:23.546411 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 155) took 3399 ms. Trace:
I20260927 19:38:23.546957 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.546805 1995 rpcz_store.cc:276] 0927 19:38:20.146594 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.480318 (+3333724us) service_pool.cc:227] Handling call
0927 19:38:23.504121 (+ 23803us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.546054 (+ 41933us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.546085 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.546133 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.546180 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.546241 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.546356 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.530553 3662 raft_consensus.cc:2866] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260927 19:38:23.548079 3542 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.541896 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.549650 1987 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:23.549775 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.550331 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1205) took 3430 ms. Trace:
I20260927 19:38:23.550992 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:23.551698 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 53537.3us
I20260927 19:38:23.552328 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:23.552589 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1214) took 3336 ms. Trace:
I20260927 19:38:23.544117 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.553161 3409 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:23.553927 2681 raft_consensus.cc:2379] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:23.554405 1993 raft_consensus.cc:1226] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:23.554886 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.555049 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1215) took 3315 ms. Trace:
I20260927 19:38:23.555426 1993 rpcz_store.cc:276] 0927 19:38:20.240161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.553442 (+3313281us) service_pool.cc:227] Handling call
0927 19:38:23.553827 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.554803 (+ 976us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.554999 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.556061 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.556710 2681 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:23.556968 1993 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:23.545050 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.558269 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.548400 3662 raft_consensus.cc:2258] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:23.550836 3416 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260927 19:38:23.559029 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 60590.2us
I20260927 19:38:23.559193 1995 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:23.559962 3584 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:23.568868 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1218) took 3302 ms. Trace:
I20260927 19:38:23.568990 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:23.569200 3584 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.569999 3558 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:23.570219 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:23.543337 2687 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:23.569272 1995 rpcz_store.cc:276] 0927 19:38:20.266921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.558300 (+3291379us) service_pool.cc:227] Handling call
0927 19:38:23.558703 (+ 403us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.568604 (+ 9901us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.568786 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.570614 1996 raft_consensus.cc:1226] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:23.571311 3545 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:23.571542 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:23.571759 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1210) took 3383 ms. Trace:
W20260927 19:38:23.541512 2709 debug-util.cc:398] Leaking SignalData structure 0x7b080035eba0 after lost signal to thread 2712
I20260927 19:38:23.572238 1995 raft_consensus.cc:2362] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4. Re-sending same reply.
I20260927 19:38:23.572440 1988 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 19:38:23.572654 2709 debug-util.cc:398] Leaking SignalData structure 0x7b08001eb340 after lost signal to thread 2713
I20260927 19:38:23.573117 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1206) took 3426 ms. Trace:
I20260927 19:38:23.553001 1989 rpcz_store.cc:276] 0927 19:38:20.215991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.551531 (+3335540us) service_pool.cc:227] Handling call
0927 19:38:23.551917 (+ 386us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.552267 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.552294 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.552339 (+ 45us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:23.552369 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.552427 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.552540 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.575841 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1219) took 3307 ms. Trace:
I20260927 19:38:23.576290 1989 rpcz_store.cc:276] 0927 19:38:20.268930 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.574570 (+3305640us) service_pool.cc:227] Handling call
0927 19:38:23.575027 (+ 457us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.575386 (+ 359us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.575426 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.575525 (+ 99us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:23.575576 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.575665 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.575783 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.562196 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.550647 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:23.560434 1962 connection.cc:232] RPC call timeout handler was delayed by -1.8831121810000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
I20260927 19:38:23.563812 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.564153 3650 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.564472 3466 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.566277 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.570427 3558 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:23.572186 1996 rpcz_store.cc:276] 0927 19:38:20.188052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.514790 (+3326738us) service_pool.cc:227] Handling call
0927 19:38:23.570130 (+ 55340us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:23.570377 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.571539 (+ 1162us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.571708 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2982272}
I20260927 19:38:23.550717 1987 rpcz_store.cc:276] 0927 19:38:20.124859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.464165 (+3339306us) service_pool.cc:227] Handling call
0927 19:38:23.500436 (+ 36271us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.550094 (+ 49658us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.550280 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.593204 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.594064 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:23.594627 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.595289 2143 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:23.595366 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:23.596051 1987 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:23.578835 2144 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:23.548388 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.596784 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:23.571803 3584 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.573565 1988 rpcz_store.cc:276] 0927 19:38:20.147194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.480785 (+3333591us) service_pool.cc:227] Handling call
0927 19:38:23.528647 (+ 47862us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.572850 (+ 44203us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.573060 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.598588 3542 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.599861 3432 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:23.600293 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:23.600950 2683 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:23.600332 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 162) took 3178 ms. Trace:
I20260927 19:38:23.601794 1988 rpcz_store.cc:276] 0927 19:38:20.422278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.598809 (+3176531us) service_pool.cc:227] Handling call
0927 19:38:23.599724 (+ 915us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.599969 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.599995 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.600056 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:23.600122 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.600181 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.600283 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.580829 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:23.572460 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:23.582159 3621 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:23.605818 3621 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.606211 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:23.607849 1995 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:23.582693 3648 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:23.617933 3648 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.582625 2654 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:23.582916 1962 connection.cc:232] RPC call timeout handler was delayed by -1.6659160689999908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2219s for a response
W20260927 19:38:23.621274 1962 connection.cc:232] RPC call timeout handler was delayed by -1.5628099450000263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.219s for a response
W20260927 19:38:23.621773 1962 connection.cc:232] RPC call timeout handler was delayed by -1.5414753930000415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260927 19:38:23.622190 1962 connection.cc:232] RPC call timeout handler was delayed by -1.2678976779999402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
W20260927 19:38:23.622568 1962 connection.cc:232] RPC call timeout handler was delayed by -1.1182175689999667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260927 19:38:23.622946 1962 connection.cc:232] RPC call timeout handler was delayed by -1.0978644679998979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2057s for a response
W20260927 19:38:23.623317 1962 connection.cc:232] RPC call timeout handler was delayed by -1.0139393679999102s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
I20260927 19:38:23.624751 1962 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:23.613319 1990 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:23.625661 3416 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:23.577859 3662 ts_tablet_manager.cc:1905] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:23.483973 3567 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:23.626425 1990 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56618 (request call id 161) took 3205 ms (client timeout 3164 ms). Trace:
W20260927 19:38:23.627413 1990 rpcz_store.cc:269] 0927 19:38:20.420509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.327594 (+2907085us) service_pool.cc:227] Handling call
0927 19:38:23.626301 (+298707us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.629732 2138 raft_consensus.cc:2396] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 19:38:23.583576 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.584709 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.585796 3494 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.586647 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:23.586773 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.588863 2139 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:38:23.592365 3617 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:23.580531 3558 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:23.600991 1987 raft_consensus.cc:1226] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:23.577797 1991 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:23.664006 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 159) took 3381 ms. Trace:
I20260927 19:38:23.664464 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1207) took 3486 ms. Trace:
I20260927 19:38:23.664570 1987 rpcz_store.cc:276] 0927 19:38:20.283159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.596763 (+3313604us) service_pool.cc:227] Handling call
0927 19:38:23.597121 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.663690 (+ 66569us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.663935 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.665180 1991 rpcz_store.cc:276] 0927 19:38:20.178494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.494655 (+3316161us) service_pool.cc:227] Handling call
0927 19:38:23.577501 (+ 82846us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.664118 (+ 86617us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.664352 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.666970 1987 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:23.668421 1991 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:23.571693 3545 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:23.669554 3545 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:23.669678 3542 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:23.670270 3542 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.671169 2141 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:23.671978 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60280 (request call id 1191) took 2995 ms. Trace:
I20260927 19:38:23.672408 1996 rpcz_store.cc:276] 0927 19:38:20.676473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.600536 (+2924063us) service_pool.cc:227] Handling call
0927 19:38:23.671678 (+ 71142us) spinlock_profiling.cc:243] Waited 67.8 ms on lock 0x7b54003606d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139687c5 00007f7913960c1a 00007f79188a46d9 00007f7911f8e184 00007f7911f8e112 00007f7911f8e041 00007f7911f8dfbc 00007f7911f8d272 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:23.671879 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155909888}
I20260927 19:38:23.603268 1994 raft_consensus.cc:595] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:23.673552 1994 raft_consensus.cc:603] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:23.580732 3574 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:23.675971 3574 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:38:23.680606 3666 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260927 19:38:23.607900 3621 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.582441 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1211) took 3394 ms. Trace:
I20260927 19:38:23.608259 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1232) took 2757 ms. Trace:
I20260927 19:38:23.608584 3560 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:23.609800 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1216) took 3344 ms. Trace:
I20260927 19:38:23.609966 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.610948 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1220) took 3341 ms. Trace:
I20260927 19:38:23.683421 1992 rpcz_store.cc:276] 0927 19:38:20.188373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.529758 (+3341385us) service_pool.cc:227] Handling call
0927 19:38:23.581738 (+ 51980us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.582079 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.582118 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.582189 (+ 71us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:23.582227 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.582286 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.582390 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.626498 3416 raft_consensus.cc:686] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:23.692503 1988 rpcz_store.cc:276] 0927 19:38:20.851081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.602802 (+2751721us) service_pool.cc:227] Handling call
0927 19:38:23.606186 (+ 3384us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.606455 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.606485 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.607991 (+ 1506us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:23.608026 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.608087 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.608192 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.661837 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 1812) took 1041 ms. Trace:
I20260927 19:38:23.694156 3560 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:23.706924 3560 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:23.677099 3574 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:23.692849 2111 leader_election.cc:334] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): Remote error: Timed out: Call waited in the queue past client deadline
W20260927 19:38:23.711661 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 101788us
W20260927 19:38:23.712051 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 129479us
I20260927 19:38:23.694691 1989 rpcz_store.cc:276] 0927 19:38:20.269241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.605455 (+3336214us) service_pool.cc:227] Handling call
0927 19:38:23.610297 (+ 4842us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.610491 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:23.610522 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:23.610690 (+ 168us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:23.610728 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.610792 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:23.610899 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.711578 2111 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:23.617944 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1234) took 2759 ms. Trace:
W20260927 19:38:23.714962 1880 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 3d1bc2cd9efe43399ddba396f5ddff85 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:23.701841 3617 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:23.713903 1995 rpcz_store.cc:276] 0927 19:38:20.858955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.607239 (+2748284us) service_pool.cc:227] Handling call
0927 19:38:23.607623 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:23.617629 (+ 10006us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.617882 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.717046 1988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:23.721508 3621 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:23.694512 1993 rpcz_store.cc:276] 0927 19:38:20.265914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.556252 (+3290338us) service_pool.cc:227] Handling call
0927 19:38:23.556591 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:23.560927 (+ 4336us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:23.609742 (+ 48815us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:23.721933 3621 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:23.722429 3621 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:23.723241 3648 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.723483 3567 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:23.724426 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 64355us
W20260927 19:38:23.731153 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 106514us
I20260927 19:38:23.737671 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:23.749702 3432 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:23.750452 3560 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.751600 1988 raft_consensus.cc:595] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:23.752087 1988 raft_consensus.cc:603] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:23.752723 3560 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.734539 3567 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:23.707893 2138 rpcz_store.cc:276] 0927 19:38:22.618579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:23.167321 (+548742us) service_pool.cc:227] Handling call
0927 19:38:23.632859 (+465538us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400225954. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:23.659870 (+ 27011us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1059261312}
I20260927 19:38:23.756026 3466 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.739799 3542 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:23.767518 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:23.768175 1996 raft_consensus.cc:595] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:23.768399 1996 raft_consensus.cc:603] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:23.761276 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:23.766685 3655 consensus_queue.cc:1046] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.005s
I20260927 19:38:23.771839 3516 raft_consensus.cc:3032] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:23.772233 3516 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:23.769641 3416 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.773190 3516 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.773939 3466 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:23.774583 3466 raft_consensus.cc:686] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:23.775686 3516 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:23.775583 3466 consensus_queue.cc:237] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.753947 3621 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.810302 1962 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.808734 3545 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:23.822243 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:23.823061 1996 raft_consensus.cc:2362] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3. Re-sending same reply.
I20260927 19:38:23.826452 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.820374 3542 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:23.839231 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.840006 2064 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:23.843662 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:23.849480 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 81735.6us
I20260927 19:38:23.857731 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.858660 2063 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:23.843787 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:38:23.874342 3655 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:23.875119 3655 raft_consensus.cc:686] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:23.816327 3432 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260927 19:38:23.885224 1988 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:38:23.865197 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:23.876353 3655 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:23.918109 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.918890 2063 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:23.919764 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 19:38:23.920217 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 52304.9us
I20260927 19:38:23.922564 3432 consensus_queue.cc:1046] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:23.923529 1962 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:23.924391 3648 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:23.925581 3564 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:23.927471 3564 consensus_queue.cc:1046] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:23.929445 3480 raft_consensus.cc:3032] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:23.930608 3432 raft_consensus.cc:3032] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:23.931947 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.933724 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.934455 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:23.939069 2064 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:23.935171 3621 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:23.939110 2068 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:23.950984 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.952731 2063 raft_consensus.cc:2396] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:23.955487 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:23.936867 3480 raft_consensus.cc:729] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:23.965653 2071 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:23.938534 3432 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:23.937534 3564 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260927 19:38:23.992376 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.993793 2063 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:23.962069 3494 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260927 19:38:23.999809 3480 consensus_queue.cc:260] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.007014 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.008785 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.009698 3574 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.017266 3480 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:24.017668 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.018989 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.019716 2063 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.020627 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:23.999692 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.015355 3558 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.021335 2063 raft_consensus.cc:2379] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.024181 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.026292 2067 raft_consensus.cc:2379] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.033020 2071 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.021617 2069 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.032773 2068 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.023135 2062 raft_consensus.cc:2396] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:24.052886 2111 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.053501 2064 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.021956 1988 raft_consensus.cc:1226] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:24.022276 3594 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260927 19:38:24.032858 3545 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.059139 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.059508 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:24.036448 3574 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.025900 3558 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.071092 1891 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:24.062897 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.067744 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.029831 3432 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.080873 2069 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.082336 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.082988 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.083391 2069 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.083648 2067 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:24.068662 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.025274 2683 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:24.069818 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.071599 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.071904 3547 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:24.092682 3547 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.094532 3594 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:24.072670 3555 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:24.073169 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.073304 3584 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:24.096300 3594 consensus_queue.cc:1046] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:24.096741 3584 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:24.096156 3555 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.097211 3584 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:24.100220 3621 consensus_queue.cc:1046] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:24.072690 3564 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:24.101296 3621 raft_consensus.cc:3032] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:24.101580 3564 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:24.101781 3621 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:24.102087 3564 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.072774 3542 consensus_queue.cc:1046] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:24.102854 3621 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.103982 3542 raft_consensus.cc:3032] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:24.104256 3621 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.104522 3542 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:24.105909 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.105762 3542 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.106635 2064 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.107560 3542 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:24.112540 3564 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.085106 2069 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:24.091234 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 152) took 6463 ms. Trace:
I20260927 19:38:24.114038 3432 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:24.118163 1959 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:24.099318 3555 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.129242 2067 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:24.079829 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.129739 1892 rpcz_store.cc:276] 0927 19:38:17.606440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.611577 (+ 5137us) service_pool.cc:227] Handling call
0927 19:38:24.069340 (+6457763us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:23.682000 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:23.706927 (+ 24927us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:23.707006 (+ 79us) write_op.cc:435] Acquired schema lock
0927 19:38:23.707026 (+ 20us) tablet.cc:660] Decoding operations
0927 19:38:23.714584 (+ 7558us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:23.714701 (+ 117us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:23.714754 (+ 53us) tablet.cc:683] Acquiring locks for 4 operations
0927 19:38:23.715258 (+ 504us) tablet.cc:699] Row locks acquired
0927 19:38:23.715291 (+ 33us) write_op.cc:260] PREPARE: finished
0927 19:38:23.715510 (+ 219us) write_op.cc:270] Start()
0927 19:38:23.715663 (+ 153us) write_op.cc:276] Timestamp: P: 1790537903715426 usec, L: 0
0927 19:38:23.715697 (+ 34us) op_driver.cc:331] REPLICATION: starting
0927 19:38:23.716408 (+ 711us) log.cc:842] Serialized 1749 byte log entry
0927 19:38:23.838959 (+122551us) op_driver.cc:444] REPLICATION: finished
0927 19:38:23.957505 (+118546us) write_op.cc:301] APPLY: starting
0927 19:38:23.957599 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:23.957792 (+ 193us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:23.957934 (+ 142us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:23.999801 (+ 41867us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:24.000132 (+ 331us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0927 19:38:24.000193 (+ 61us) write_op.cc:312] APPLY: finished
0927 19:38:24.000500 (+ 307us) log.cc:842] Serialized 41 byte log entry
0927 19:38:24.000682 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:24.000787 (+ 105us) write_op.cc:454] Released schema lock
0927 19:38:24.000857 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:24.001190 (+ 333us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:24.001229 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":551752,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":113505,"apply.run_cpu_time_us":3639,"apply.run_wall_time_us":44072,"num_ops":4,"prepare.queue_time_us":173854,"prepare.run_cpu_time_us":13645,"prepare.run_wall_time_us":64633,"raft.queue_time_us":76839,"raft.run_cpu_time_us":30314,"raft.run_wall_time_us":79529,"replication_time_us":119877,"thread_start_us":201927,"threads_started":4,"wal-append.queue_time_us":30261}]]}
I20260927 19:38:24.141848 3368 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:24.142441 3547 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.142732 3480 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:24.130720 3684 consensus_queue.cc:1046] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260927 19:38:24.143931 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.146026 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:24.162832 1962 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.172266 2679 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:24.172827 3584 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.173684 2069 raft_consensus.cc:2379] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.174136 1988 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:24.174516 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.165059 3368 raft_consensus.cc:686] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:24.174784 3684 raft_consensus.cc:3032] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:24.177301 3368 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.182682 3584 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.184175 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:24.186096 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:24.177601 3684 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:24.187213 3684 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.188591 3684 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:24.179479 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 47636.2us
I20260927 19:38:24.199010 3567 raft_consensus.cc:3032] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:24.203065 1988 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:24.203644 3693 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.003s
I20260927 19:38:24.132537 3662 ts_tablet_manager.cc:1918] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260927 19:38:24.229897 3662 log.cc:1196] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-1-root/wals/0fcc7a171b3e443d8f28fd70f8742470
I20260927 19:38:24.229965 3693 raft_consensus.cc:3032] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:24.230490 3693 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:24.231392 3662 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48542 (request call id 855) took 1509 ms. Trace:
I20260927 19:38:24.231863 3650 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.233961 3691 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:24.231439 3693 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.237465 3693 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:24.214731 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 76920.3us
I20260927 19:38:24.216320 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.241745 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:24.241920 2069 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.242874 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.243450 2069 raft_consensus.cc:2362] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2. Re-sending same reply.
I20260927 19:38:24.243901 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.244211 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.246125 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.216596 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.246814 3567 raft_consensus.cc:729] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:24.247218 2063 raft_consensus.cc:2379] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.217911 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.218120 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.218477 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.216730 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.219638 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.220402 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.248896 2070 raft_consensus.cc:2396] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:24.249425 2068 raft_consensus.cc:2396] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:24.253711 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.248799 2064 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.219421 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 254) took 6584 ms. Trace:
I20260927 19:38:24.244931 2136 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:24.242882 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.247969 2066 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.223662 2065 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.264461 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.264966 2145 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:24.267119 2063 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.269877 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:24.270282 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 63920.2us
I20260927 19:38:24.273340 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.273795 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.272574 2067 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.253911 3564 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.277226 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.279158 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.280018 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.265425 2062 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.288120 3703 consensus_queue.cc:1046] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260927 19:38:24.283192 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.277166 3621 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:24.293280 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.293641 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.295840 2143 raft_consensus.cc:2379] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:24.255769 1891 rpcz_store.cc:276] 0927 19:38:17.635353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:17.635492 (+ 139us) service_pool.cc:227] Handling call
0927 19:38:24.219358 (+6583866us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:24.091821 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:24.143212 (+ 51391us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:24.143247 (+ 35us) write_op.cc:435] Acquired schema lock
0927 19:38:24.143264 (+ 17us) tablet.cc:660] Decoding operations
0927 19:38:24.143632 (+ 368us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:24.143688 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:24.143723 (+ 35us) tablet.cc:683] Acquiring locks for 1 operations
0927 19:38:24.143934 (+ 211us) tablet.cc:699] Row locks acquired
0927 19:38:24.143953 (+ 19us) write_op.cc:260] PREPARE: finished
0927 19:38:24.144107 (+ 154us) write_op.cc:270] Start()
0927 19:38:24.144224 (+ 117us) write_op.cc:276] Timestamp: P: 1790537904144064 usec, L: 0
0927 19:38:24.144249 (+ 25us) op_driver.cc:331] REPLICATION: starting
0927 19:38:24.144873 (+ 624us) log.cc:842] Serialized 536 byte log entry
0927 19:38:24.147185 (+ 2312us) op_driver.cc:444] REPLICATION: finished
0927 19:38:24.147676 (+ 491us) write_op.cc:301] APPLY: starting
0927 19:38:24.147767 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:24.147877 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:24.148001 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:24.148281 (+ 280us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:24.148439 (+ 158us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0927 19:38:24.148493 (+ 54us) write_op.cc:312] APPLY: finished
0927 19:38:24.148686 (+ 193us) log.cc:842] Serialized 23 byte log entry
0927 19:38:24.148882 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:24.148983 (+ 101us) write_op.cc:454] Released schema lock
0927 19:38:24.149073 (+ 90us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:24.149441 (+ 368us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:24.149482 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4195764,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":231,"apply.run_cpu_time_us":2070,"apply.run_wall_time_us":2069,"num_ops":1,"prepare.queue_time_us":9416,"prepare.run_cpu_time_us":2692,"prepare.run_wall_time_us":53363,"replication_time_us":2831}]]}
I20260927 19:38:24.231834 3662 rpcz_store.cc:276] 0927 19:38:22.722457 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:22.722653 (+ 196us) service_pool.cc:227] Handling call
0927 19:38:22.922047 (+199394us) ts_tablet_manager.cc:1145] Deleting tablet 0fcc7a171b3e443d8f28fd70f8742470
0927 19:38:22.922110 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0927 19:38:23.375591 (+453481us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400212ad4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f791396a352 00007f791733226c 00007f79188f2cfa 00007f7918922e10 00007f7918909e0d 00007f7918909db9 00007f7918909d49 00007f7918909d11 00007f7918908f3d 00007f7919c9e204
0927 19:38:23.753186 (+377595us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:24.092085 (+338899us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:24.231287 (+139202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30381,"spinlock_wait_cycles":768326016,"tablet-delete.queue_time_us":13051,"thread_start_us":30700,"threads_started":1}
I20260927 19:38:24.293538 3621 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:24.301789 3621 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:24.295565 2136 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.299247 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.306564 2062 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.300209 3480 raft_consensus.cc:3032] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:24.247951 3567 consensus_queue.cc:260] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.307389 3480 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:24.304540 1879 catalog_manager.cc:4841] TS 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): tablet 0fcc7a171b3e443d8f28fd70f8742470 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:24.291195 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.308734 2138 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:24.309116 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.310068 2062 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.310612 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.310966 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.311275 2063 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:24.311285 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.312855 2067 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.312673 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.313134 3594 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:24.308753 3480 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.313963 2071 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:24.315718 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.316309 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.315739 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.314494 3480 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.318010 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.318693 1996 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:24.323796 1991 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:24.309075 1962 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.327750 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.330134 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:24.308054 3567 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:24.330387 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.332386 3494 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:24.332780 3494 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:24.333885 3494 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:24.334841 2138 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:24.337615 2654 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.340601 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.341306 3621 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.341969 3494 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.342710 1990 raft_consensus.cc:2354] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5.
I20260927 19:38:24.356104 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.357625 3494 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.358956 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.359977 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.360869 1990 raft_consensus.cc:2379] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:24.358987 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.362119 1987 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:24.364419 3542 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.365258 2681 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:24.374492 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.375231 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.345700 1962 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:24.394872 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 49259.2us
I20260927 19:38:24.395830 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.397476 3650 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.397886 3584 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:24.398190 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.399991 1962 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.402328 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:24.376734 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45523us
I20260927 19:38:24.406420 3691 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:24.407277 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.377689 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.378300 3693 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:24.410096 2136 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:24.410311 3693 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:24.410813 3693 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.411460 2108 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.412827 3480 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:24.413298 3480 raft_consensus.cc:686] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:24.414507 3480 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.380419 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.380959 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.381847 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.383469 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.385598 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.343540 3621 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.388404 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.392153 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.402978 3650 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:24.406693 3334 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:24.431092 3334 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:24.431649 3334 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:24.431751 3542 raft_consensus.cc:979] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:24.346012 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.423815 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.429672 3650 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:24.435364 3650 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.435451 1995 raft_consensus.cc:2396] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:24.438885 3703 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:24.439304 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.440037 2136 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:24.440471 3703 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:24.378990 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.451541 1996 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:24.455019 1891 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 3 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:24.464144 1891 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:24.467216 1891 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:24.444347 3621 raft_consensus.cc:1067] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:24.474159 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.470175 1891 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:24.486266 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.487409 1989 raft_consensus.cc:2396] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:24.487502 3693 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.488315 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.489617 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.494529 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.498541 2143 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:24.532646 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 66800.5us
I20260927 19:38:24.490193 3693 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.543813 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.543871 3703 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:24.544543 3703 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.546584 3703 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.527233 1891 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 3 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:24.547669 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:24.552541 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.553284 2144 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.552240 1891 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:24.554800 1891 catalog_manager.cc:5495] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 2, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 2 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:24.561740 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.564145 2144 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:24.572197 2141 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260927 19:38:24.574849 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.576918 2139 raft_consensus.cc:1226] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:24.572618 2141 pending_rounds.cc:76] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 4
I20260927 19:38:24.578564 2141 pending_rounds.cc:97] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260927 19:38:24.486295 3334 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.582981 2145 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:38:24.566103 3693 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:24.583704 2141 raft_consensus.cc:2866] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:24.584537 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.594931 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.596545 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.602319 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.606673 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.607749 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.582861 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.556892 1891 catalog_manager.cc:5495] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:24.612156 2144 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:24.613970 1992 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
W20260927 19:38:24.629498 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 58234.9us
I20260927 19:38:24.631738 2111 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.616083 3334 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:24.616642 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.641240 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.616971 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.642127 3703 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:24.642504 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.643571 2683 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:24.643991 3648 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:24.581887 3650 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.644629 2111 leader_election.cc:398] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5.
I20260927 19:38:24.645041 2111 leader_election.cc:401] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260927 19:38:24.645815 2111 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:24.646100 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.646327 3650 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.617441 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.649063 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.623986 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.642369 2139 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:24.581547 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:24.644577 3648 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:24.647828 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.626354 2136 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:24.650017 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.656085 2144 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.628346 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.643970 3703 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:24.659550 1961 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.659776 2111 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.678711 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.649854 1989 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:24.669893 2654 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.681080 1992 raft_consensus.cc:2379] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:24.683684 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.685139 2144 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:24.642124 1987 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.650209 3555 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.689095 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.688680 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.664084 3480 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5.
I20260927 19:38:24.657673 3648 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.650297 1993 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:24.650393 3574 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260927 19:38:24.698599 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:24.690343 3690 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:24.664283 3650 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260927 19:38:24.705826 3516 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:24.707645 3584 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:24.708791 1962 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.709306 3621 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:24.709527 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.710325 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.711530 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.707818 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.658334 3684 consensus_queue.cc:1046] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20260927 19:38:24.720266 2139 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:24.748075 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:24.739672 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.749611 1992 raft_consensus.cc:2379] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:24.744684 3334 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:24.738751 3584 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:24.761956 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.763904 1992 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:24.762844 2679 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260927 19:38:24.745573 3690 raft_consensus.cc:686] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:24.756021 3432 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:24.781227 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.782622 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:24.783941 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.784543 1992 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:24.787969 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.788728 1990 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:24.765801 3584 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.801383 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.771831 3650 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:24.769896 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:24.815610 3621 raft_consensus.cc:686] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:24.819139 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 111402us
W20260927 19:38:24.832345 2037 connection.cc:232] RPC call timeout handler was delayed by -1.9565008549999447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260927 19:38:24.841679 2037 connection.cc:232] RPC call timeout handler was delayed by -1.8476008549998824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
I20260927 19:38:24.842010 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.815377 3334 raft_consensus.cc:686] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:24.845906 2037 connection.cc:232] RPC call timeout handler was delayed by -1.7531008549999569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
W20260927 19:38:24.847378 2037 connection.cc:232] RPC call timeout handler was delayed by -1.7198008549999031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260927 19:38:24.847822 2037 connection.cc:232] RPC call timeout handler was delayed by -1.6343008549998785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260927 19:38:24.848277 2037 connection.cc:232] RPC call timeout handler was delayed by -1.6001008549999369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
W20260927 19:38:24.848677 2037 connection.cc:232] RPC call timeout handler was delayed by -1.5731008549998933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
I20260927 19:38:24.850157 3334 consensus_queue.cc:237] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.860895 3729 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:24.864497 3648 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.841329 3691 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260927 19:38:24.869592 3690 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.870846 3686 raft_consensus.cc:3032] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
W20260927 19:38:24.849136 2037 connection.cc:232] RPC call timeout handler was delayed by -1.5641008549998787s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19870000000000002s for a response
W20260927 19:38:24.881883 2037 connection.cc:232] RPC call timeout handler was delayed by -1.5380008549999502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260927 19:38:24.882406 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4678008549999504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260927 19:38:24.882845 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4381008549999024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
W20260927 19:38:24.883270 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4174008549999826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
W20260927 19:38:24.883702 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4156008549999797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
W20260927 19:38:24.884285 2037 connection.cc:232] RPC call timeout handler was delayed by -1.408400854999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20260927 19:38:24.884796 2037 connection.cc:232] RPC call timeout handler was delayed by -1.3958008549999477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
W20260927 19:38:24.886543 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 138563us
W20260927 19:38:24.885229 2037 connection.cc:232] RPC call timeout handler was delayed by -1.1258008549999658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24740000000000004s for a response
I20260927 19:38:24.837953 3650 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:24.898162 3650 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.900996 1992 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:24.902884 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.903283 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.903569 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.903308 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.904881 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.903914 2067 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:24.878360 3686 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:24.911613 2064 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:24.910343 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.910683 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.914438 2681 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:24.890412 1987 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:24.917112 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.917935 2062 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:24.918229 3368 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.919719 3648 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:24.921170 3542 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:24.921646 3542 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:24.922505 3542 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.923782 3542 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:24.908849 1989 raft_consensus.cc:1226] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:24.932402 1992 raft_consensus.cc:1261] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:24.888502 3734 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:24.936232 2068 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.936640 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.941587 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:24.941682 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.942909 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.943122 2067 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:24.950080 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:24.950903 2070 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 3
I20260927 19:38:24.935405 3547 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:24.962147 2068 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:24.941766 3728 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.928097 3724 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
W20260927 19:38:24.966379 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 124482us
I20260927 19:38:24.957700 3729 raft_consensus.cc:686] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:24.938226 3545 consensus_queue.cc:1046] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260927 19:38:24.964427 2138 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260927 19:38:24.972852 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.973887 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.973809 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:24.970130 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 54817.3us
I20260927 19:38:24.970882 1961 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:24.975456 2071 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:24.976830 2068 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:24.976847 2069 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:24.980620 3741 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:24.981037 3741 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:24.938444 3621 consensus_queue.cc:237] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.983170 3729 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.963868 3686 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:24.981446 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:24.965457 3734 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.996111 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:24.998773 3334 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:24.999043 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:25.000177 1961 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.001597 3368 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.002224 3368 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:25.002569 3368 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.002951 3368 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:24.996610 3584 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:24.981403 3741 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:25.016366 1995 raft_consensus.cc:1226] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:24.995225 3734 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.018303 2070 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:25.023867 1961 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.025463 2063 raft_consensus.cc:1226] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 3. Ops: []
I20260927 19:38:25.028865 1961 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.030228 1961 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.031069 3724 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:25.019222 3739 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260927 19:38:25.039753 1961 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.042115 2068 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:25.043673 2064 raft_consensus.cc:1226] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.046190 2069 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:25.032039 3621 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:25.049512 2069 raft_consensus.cc:1226] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.049809 1996 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 19:38:25.035360 3724 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.055630 2062 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.045559 3648 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:25.057117 3648 raft_consensus.cc:686] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:25.057559 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:25.058176 3648 consensus_queue.cc:237] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.059161 2067 raft_consensus.cc:1226] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.064181 3650 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.050976 3737 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.066088 3584 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.065450 3724 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.067893 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.069676 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.073493 3432 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:25.074211 3432 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.076578 3432 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:25.079938 3730 raft_consensus.cc:3032] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:25.080428 3730 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:25.081483 3730 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.082703 3730 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:25.083874 3724 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:25.092610 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52952.1us
I20260927 19:38:25.095229 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.097956 1961 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.099761 2069 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:25.101943 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.107390 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:25.108909 2653 leader_election.cc:334] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.513s (SENT)
I20260927 19:38:25.112598 2063 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 19:38:25.112699 2067 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 19:38:25.108274 2683 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:25.116029 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.116374 2679 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:25.116976 3724 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.117949 2108 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.119598 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:25.120805 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:25.121812 3547 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.122319 3547 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.122777 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.122978 2068 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:25.124294 2066 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 19:38:25.099785 3594 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:25.125222 3594 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.125648 3703 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:25.126168 3703 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:25.126631 3703 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:25.102837 3737 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.128582 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.130662 2067 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:25.133229 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.136103 2069 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:25.139252 1961 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:25.139946 2110 leader_election.cc:334] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.943s (SENT)
I20260927 19:38:25.140450 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.142145 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:25.143814 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:25.144519 2071 raft_consensus.cc:2362] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4. Re-sending same reply.
I20260927 19:38:25.145488 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.146128 2071 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:25.117743 3684 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.149533 3684 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.150832 3621 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.152570 3684 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.133464 3432 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260927 19:38:25.155659 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.157015 3686 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:25.138248 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.161437 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.163084 2066 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:25.174525 2110 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:25.124312 3547 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:25.134051 3594 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:25.187549 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:25.172866 3724 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:25.195775 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 55423.5us
I20260927 19:38:25.198032 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.198771 2068 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:25.188835 1995 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:25.193419 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.196928 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.203250 1961 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.200169 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.192408 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:25.127712 3368 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.189172 3584 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:25.209403 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.211859 1961 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.194121 3432 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:25.221413 3740 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20260927 19:38:25.222934 3745 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260927 19:38:25.224993 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 85077.2us
I20260927 19:38:25.209028 3724 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.217120 3729 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:25.246083 3729 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.246516 3729 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.214605 3730 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.212291 3584 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.211802 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:25.245297 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.252828 2138 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:25.254136 3730 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:25.254772 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.255373 3737 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.256583 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.252228 3584 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:25.231073 3650 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.269079 3368 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.248170 3334 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:25.271901 2654 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.278703 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.273466 3334 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.280808 3334 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.293416 3594 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.293457 3684 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:25.294514 3684 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:25.295301 3684 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:25.294523 3594 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.300444 3594 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:25.317721 3745 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:25.318244 3745 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:25.318625 3745 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.322434 3741 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.324225 2137 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:25.325759 3655 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:25.325874 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.326746 2137 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:25.328074 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 73455us
W20260927 19:38:25.330798 2653 leader_election.cc:334] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.736s (SENT)
I20260927 19:38:25.331286 2653 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:25.332535 3691 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.240948 1961 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:25.344311 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 103452us
I20260927 19:38:25.345263 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.346745 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:25.362560 2653 leader_election.cc:334] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.768s (SENT)
I20260927 19:38:25.363279 2653 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.372918 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:25.348909 3368 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:25.379669 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.359879 3545 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.376958 3621 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.383973 3494 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.288914 3584 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.387398 3368 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.415550 3368 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:25.417549 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 70865us
I20260927 19:38:25.418399 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.419183 3545 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.393175 3737 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.360195 3703 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.447166 3703 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:25.411744 3724 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:25.449615 3724 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:25.450070 3724 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:25.450776 3368 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.458315 3729 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.460723 3729 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.463234 3684 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.393843 3741 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:25.433487 3737 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.436513 2137 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:25.376401 3745 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.473786 3684 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.474992 3745 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.441668 1961 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:25.477137 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 53551.3us
I20260927 19:38:25.477416 3584 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.478091 3584 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:25.478439 3584 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:25.478770 3584 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.480845 2111 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.482784 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.484792 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.485598 3594 raft_consensus.cc:2741] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 19:38:25.426904 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:25.486138 2682 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260927 19:38:25.486308 2681 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:25.486562 2682 pending_rounds.cc:76] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Aborting all ops after (but not including) 6
I20260927 19:38:25.486920 2682 pending_rounds.cc:97] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260927 19:38:25.487205 2682 raft_consensus.cc:2866] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:25.464303 3368 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.423828 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:25.503432 1996 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:25.505311 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.508059 1995 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.511049 2140 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260927 19:38:25.514446 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 90666.1us
I20260927 19:38:25.518033 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:25.518136 3736 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260927 19:38:25.524101 3737 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.509544 3745 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:25.562431 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 153) took 1144 ms. Trace:
I20260927 19:38:25.567337 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.537671 3734 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.572180 1996 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:25.555096 3745 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:25.553673 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:25.581841 3734 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.570336 3737 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.593441 3734 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.593865 3730 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260927 19:38:25.584038 3545 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.565418 3735 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260927 19:38:25.598012 1992 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.663602 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:25.666919 1996 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:25.639473 1994 raft_consensus.cc:1226] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:25.657853 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:25.668895 2139 raft_consensus.cc:2396] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:25.684302 2683 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:25.570956 1891 rpcz_store.cc:276] 0927 19:38:24.409665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:24.410108 (+ 443us) service_pool.cc:227] Handling call
0927 19:38:25.553541 (+1143433us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:24.905425 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:24.925046 (+ 19621us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:24.925088 (+ 42us) write_op.cc:435] Acquired schema lock
0927 19:38:24.925105 (+ 17us) tablet.cc:660] Decoding operations
0927 19:38:24.926002 (+ 897us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:24.926058 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:24.926100 (+ 42us) tablet.cc:683] Acquiring locks for 8 operations
0927 19:38:24.926714 (+ 614us) tablet.cc:699] Row locks acquired
0927 19:38:24.926735 (+ 21us) write_op.cc:260] PREPARE: finished
0927 19:38:24.926874 (+ 139us) write_op.cc:270] Start()
0927 19:38:24.926981 (+ 107us) write_op.cc:276] Timestamp: P: 1790537904926835 usec, L: 0
0927 19:38:24.927002 (+ 21us) op_driver.cc:331] REPLICATION: starting
0927 19:38:24.965829 (+ 38827us) log.cc:842] Serialized 3218 byte log entry
0927 19:38:25.109542 (+143713us) op_driver.cc:444] REPLICATION: finished
0927 19:38:25.381937 (+272395us) write_op.cc:301] APPLY: starting
0927 19:38:25.382024 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:25.382218 (+ 194us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:25.382332 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:25.383378 (+ 1046us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:25.383572 (+ 194us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0927 19:38:25.383612 (+ 40us) write_op.cc:312] APPLY: finished
0927 19:38:25.383873 (+ 261us) log.cc:842] Serialized 65 byte log entry
0927 19:38:25.384043 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:25.384161 (+ 118us) write_op.cc:454] Released schema lock
0927 19:38:25.384207 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:25.384436 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:25.384465 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5402,"reported_tablets":16,"child_traces":[["op",{"apply.queue_time_us":262350,"apply.run_cpu_time_us":2829,"apply.run_wall_time_us":2830,"num_ops":8,"prepare.queue_time_us":186480,"prepare.run_cpu_time_us":4043,"prepare.run_wall_time_us":90988,"replication_time_us":172947,"thread_start_us":340405,"threads_started":2}]]}
W20260927 19:38:25.694830 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 298626us
I20260927 19:38:25.697542 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:25.700696 2653 leader_election.cc:334] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.817s (SENT)
I20260927 19:38:25.700950 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:25.702937 1994 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:25.633955 1993 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:25.593937 3745 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:25.704282 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:25.725488 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 223689us
I20260927 19:38:25.711442 1892 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 3, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:25.594702 1989 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:25.700429 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:25.717558 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 179849us
I20260927 19:38:25.728749 1992 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:25.732784 3494 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:25.731628 1892 catalog_manager.cc:5495] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.743870 1892 catalog_manager.cc:5495] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.750643 1892 catalog_manager.cc:5495] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.756850 1892 catalog_manager.cc:5495] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.762744 1892 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 3, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 3 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:25.613445 3737 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.773528 1987 raft_consensus.cc:1226] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.795993 1892 catalog_manager.cc:5495] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.811817 1892 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.814991 2139 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:38:25.806281 3776 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:25.812654 3542 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:25.821810 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 152056us
I20260927 19:38:25.826747 2144 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:25.820869 1892 catalog_manager.cc:5495] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
W20260927 19:38:25.831768 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 105435us
I20260927 19:38:25.832969 1892 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 4, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.834849 1892 catalog_manager.cc:5495] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 2 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:25.842126 2108 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.824186 3735 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260927 19:38:25.838768 2138 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 19:38:25.851356 3334 consensus_meta.cc:287] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.570s user 0.012s sys 0.000s
I20260927 19:38:25.851822 3334 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.852463 2681 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:25.853053 1988 raft_consensus.cc:1226] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:25.852854 3741 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:25.835773 3432 consensus_queue.cc:1046] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260927 19:38:25.842478 1991 raft_consensus.cc:1226] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:25.854650 3724 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.825664 3542 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.822094 3776 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.868274 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.834453 3650 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260927 19:38:25.836695 3703 consensus_queue.cc:1046] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260927 19:38:25.875349 3784 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:25.877352 3737 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.878038 3584 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:25.879616 1995 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 19:38:25.870215 3783 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:25.883276 3776 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:25.887197 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:25.893795 2144 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:25.863600 3724 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.901235 1962 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.844695 3777 consensus_queue.cc:1046] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
W20260927 19:38:25.914155 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 346906us
I20260927 19:38:25.915012 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.918093 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.904320 3724 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:25.925592 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:25.912995 3772 raft_consensus.cc:3032] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:25.929620 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.940171 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:25.954982 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 68052.9us
I20260927 19:38:25.954799 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:25.953939 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:25.923261 3334 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.926151 3724 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:25.916523 3776 consensus_queue.cc:1046] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260927 19:38:25.932278 3779 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.929461 3772 raft_consensus.cc:729] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:25.970286 2144 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:25.932715 3545 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:25.973686 1962 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.928866 3542 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:25.973001 2111 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:25.942585 3730 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:25.967375 3734 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:25.972930 3724 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:25.994933 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:26.029838 2144 raft_consensus.cc:595] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:25.974725 3545 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:26.031400 3545 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:25.988366 3779 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:26.000029 2110 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.813s (SENT) [suppressed 61 similar messages]
I20260927 19:38:26.000070 3432 consensus_queue.cc:1046] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:26.001662 3547 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:26.003274 3786 raft_consensus.cc:3032] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:25.991016 3783 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.013862 3772 consensus_queue.cc:260] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.043617 3772 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.015645 3584 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.017089 3574 raft_consensus.cc:3032] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:26.045749 3574 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.046738 3574 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.018908 1989 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:25.967563 3745 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:25.986359 3334 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:26.022522 3794 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:26.048512 1961 leader_election.cc:334] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.219s (SENT)
I20260927 19:38:26.055691 1961 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.039768 3547 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:26.047727 3776 raft_consensus.cc:3032] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.048046 3574 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.050256 3334 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:26.041138 3786 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.050081 3745 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.076321 3735 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:26.078461 3547 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.078019 3776 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.078896 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:26.091715 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 136997us
I20260927 19:38:26.084240 3334 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:26.085826 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:26.103821 2037 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.104882 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:26.063719 3794 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:26.112260 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 139350us
W20260927 19:38:26.117489 3781 log.cc:924] Time spent T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.124s user 0.000s sys 0.002s
I20260927 19:38:26.124217 3545 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:26.110795 2037 raft_consensus.cc:2670] T 0fcc7a171b3e443d8f28fd70f8742470 P 4cc2df048f424e08803a390946004df0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260927 19:38:26.133121 1961 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.149991 2653 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.112176 1989 raft_consensus.cc:595] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:26.154577 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:26.155181 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 155171us
I20260927 19:38:26.156826 2144 raft_consensus.cc:1261] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260927 19:38:26.158075 2144 pending_rounds.cc:76] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 1
I20260927 19:38:26.158306 2144 pending_rounds.cc:97] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:38:26.158526 2144 raft_consensus.cc:2866] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 19:38:26.164206 2110 leader_election.cc:334] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.792s (SENT)
I20260927 19:38:26.166577 1961 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.168751 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:26.103551 3783 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.171066 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:26.172852 1961 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.155326 3741 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:26.162736 3791 raft_consensus.cc:3032] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.177165 2063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:26.176520 2653 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:26.158828 3368 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.180742 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.186537 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.186859 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.187294 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:26.188125 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.178685 3741 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:26.188834 3741 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.181044 3791 raft_consensus.cc:729] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.190778 3791 consensus_queue.cc:260] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.192247 3791 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.182729 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.193542 3747 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:26.193988 3545 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.183422 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.194823 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.195480 2067 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:26.181272 2063 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:26.196259 2063 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.173251 3778 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.184134 3542 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.183604 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:26.114109 3786 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.179273 2144 raft_consensus.cc:1226] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:26.173810 3801 consensus_queue.cc:1046] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260927 19:38:26.200326 3786 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.185063 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:26.197167 3774 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:26.129428 3776 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.201402 1990 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:26.153505 3795 consensus_queue.cc:1046] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260927 19:38:26.204149 3795 raft_consensus.cc:3032] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.204526 3795 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.205286 3795 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.206595 3795 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.189622 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.209766 2070 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:26.211217 3542 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.213516 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:26.197412 3778 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.217972 3778 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.218107 3334 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.176429 3784 consensus_queue.cc:1046] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260927 19:38:26.219177 3778 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.219664 3784 raft_consensus.cc:3032] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:26.217005 1988 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:26.220078 3784 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.213871 3547 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.222286 3547 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:26.223734 3774 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.224100 3774 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.224156 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:26.226008 2071 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:26.224908 3774 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.229050 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.231611 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83f97251c21d4e979bce8419c208369c"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:26.231775 2141 raft_consensus.cc:1203] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-4.3] Dedup: 4.3->[]
I20260927 19:38:26.232246 2141 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 19:38:26.232246 2145 raft_consensus.cc:595] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:26.168817 3655 raft_consensus.cc:2741] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 19:38:26.233431 2145 raft_consensus.cc:603] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 19:38:26.238143 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 55311us
I20260927 19:38:26.189218 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:26.246869 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 74061.3us
I20260927 19:38:26.174554 3802 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260927 19:38:26.227316 3796 consensus_queue.cc:1046] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260927 19:38:26.247856 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.213148 3792 raft_consensus.cc:3032] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:26.255187 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:26.177215 3584 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.266151 2064 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:26.257714 2066 raft_consensus.cc:3032] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:26.266525 2066 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
W20260927 19:38:26.259064 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 74069.1us
I20260927 19:38:26.262194 2067 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:26.268034 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.268867 2067 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.275925 1995 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:26.277848 1996 raft_consensus.cc:1226] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:26.274906 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.281661 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:26.282485 2068 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:26.285073 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.178413 3799 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.290904 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.292754 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:26.296038 2145 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:26.249217 3774 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.298791 2037 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:26.293121 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:26.289781 2066 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.304131 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:26.307976 2067 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:26.301733 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 114375us
I20260927 19:38:26.309628 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:26.310302 1994 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.312278 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:26.312949 2141 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:26.313798 2066 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:26.265781 3792 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.293622 3776 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.320490 2070 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:26.323333 2653 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.284253 3728 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.321768 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.220930 3784 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.291725 3334 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.330511 2139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "901577b40c874966936247e62860f154"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:26.248905 3368 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.271335 3793 consensus_queue.cc:1046] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260927 19:38:26.341240 3727 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.341533 3730 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.328996 3745 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:26.377557 1994 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:26.346393 3794 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.349496 3797 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.350878 3741 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.360414 2145 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:26.334111 3686 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:26.329454 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:26.370587 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 48887.6us
I20260927 19:38:26.360190 2063 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.373919 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:26.378985 2111 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:26.380584 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46988.8us
I20260927 19:38:26.333124 3728 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.302423 3799 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.368117 3802 consensus_queue.cc:1046] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
W20260927 19:38:26.406776 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 52504.7us
I20260927 19:38:26.423871 2141 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:26.424131 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 45212.9us
I20260927 19:38:26.401466 1962 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:26.428151 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:26.430682 2063 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:26.412537 3805 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260927 19:38:26.416857 3780 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260927 19:38:26.381227 3745 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:26.376103 2062 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260927 19:38:26.409183 1988 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:26.413504 3784 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:26.485149 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 55895.6us
I20260927 19:38:26.468901 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:26.468634 2064 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:26.406553 3334 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.430193 3545 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.407714 3368 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.469098 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:26.368391 3724 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.403295 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:26.431891 3735 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:26.409042 3574 raft_consensus.cc:3032] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.377719 3776 consensus_queue.cc:1046] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20260927 19:38:26.439395 3794 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:26.485137 3795 raft_consensus.cc:3032] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
W20260927 19:38:26.524376 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 55368us
I20260927 19:38:26.536475 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.438972 1987 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:26.507019 2062 pending_rounds.cc:76] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 4
W20260927 19:38:26.530378 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 104591us
I20260927 19:38:26.460121 3734 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.506486 3745 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.525543 3368 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.528666 3730 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:26.493456 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:26.432129 3801 consensus_queue.cc:1046] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20260927 19:38:26.540161 2654 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:26.535035 3735 raft_consensus.cc:686] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:26.551270 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 126973us
I20260927 19:38:26.528645 3574 raft_consensus.cc:729] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.379439 3727 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.538252 2062 pending_rounds.cc:97] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260927 19:38:26.548892 3777 raft_consensus.cc:3032] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:26.401724 3797 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.527113 3802 consensus_queue.cc:1046] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260927 19:38:26.536031 3795 raft_consensus.cc:729] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.523499 3334 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.564399 2066 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260927 19:38:26.575392 3794 raft_consensus.cc:686] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.565413 2062 raft_consensus.cc:2866] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:26.576432 3813 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260927 19:38:26.577942 2139 raft_consensus.cc:595] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:26.586141 2139 raft_consensus.cc:603] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:26.571820 3777 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.579464 3797 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:26.581434 2066 pending_rounds.cc:76] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 2
I20260927 19:38:26.593698 2066 pending_rounds.cc:97] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 19:38:26.558214 3734 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:26.594020 2066 raft_consensus.cc:2866] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:26.594378 3734 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.595624 3777 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.596411 3734 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.597200 3777 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.597672 1991 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:26.567891 3796 consensus_queue.cc:1046] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260927 19:38:26.579553 3735 consensus_queue.cc:237] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.602033 3795 consensus_queue.cc:260] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.603119 3795 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:26.606308 3728 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.567521 3730 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:26.620543 1995 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
W20260927 19:38:26.642251 3814 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260927 19:38:26.643714 2681 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:26.644662 3574 consensus_queue.cc:260] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.608800 3779 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:26.651763 3794 consensus_queue.cc:237] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.652521 3779 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.619199 3594 consensus_queue.cc:1046] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260927 19:38:26.558374 3368 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.656227 3368 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.656759 3334 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.648747 3727 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.650317 3741 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.651132 3724 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.665649 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:26.631942 3772 consensus_meta.cc:287] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.588s user 0.013s sys 0.000s
I20260927 19:38:26.683341 3799 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:26.631603 3813 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:26.690613 3813 raft_consensus.cc:686] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:26.691944 3813 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.693545 3800 raft_consensus.cc:3032] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:26.694096 3800 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:26.695190 3800 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.703049 1959 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.771179 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:26.771859 2141 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:26.774290 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 71381.4us
I20260927 19:38:26.777282 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:26.779863 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:26.780546 2679 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:26.795054 3779 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.800184 2141 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
W20260927 19:38:26.801844 3778 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.582s user 0.012s sys 0.010s
I20260927 19:38:26.808981 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:26.809991 2683 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:26.819048 1959 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:26.820399 1959 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:26.822388 3542 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:26.822855 3542 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:26.823264 3542 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.824307 3745 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.826351 3724 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:26.826821 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:26.837633 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:26.842388 3542 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:26.871025 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:26.878607 1962 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:26.894925 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 336957us
W20260927 19:38:26.902361 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 365951us
I20260927 19:38:26.896341 3784 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:26.918037 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:26.901994 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 347026us
I20260927 19:38:26.919270 2111 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:26.913844 2141 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:26.922593 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44100.6us
I20260927 19:38:26.923888 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:26.925000 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:26.901995 3724 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:26.945333 1962 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:26.960134 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:26.964159 2685 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:26.932396 2110 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:26.936941 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:26.941990 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:26.970566 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 413268us
W20260927 19:38:26.932853 3815 log.cc:924] Time spent T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.265s user 0.001s sys 0.009s
I20260927 19:38:26.922821 1990 raft_consensus.cc:595] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:38:26.929034 3827 log.cc:924] Time spent T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260927 19:38:26.928161 3799 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.928041 3368 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:26.955608 3779 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:26.959079 3545 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:26.935159 3724 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:26.932680 3778 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:26.998425 3778 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:26.998812 3778 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:26.999373 3368 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:27.000432 3368 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:26.955360 3777 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20260927 19:38:26.981331 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 49047.6us
I20260927 19:38:26.982333 3825 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:26.982583 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:26.982918 3829 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:26.983027 1991 raft_consensus.cc:1226] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:26.980967 3432 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:26.984972 3830 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:26.984974 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:26.986251 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:26.992180 1987 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:27.013557 2141 raft_consensus.cc:1261] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:27.015247 2685 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:27.015288 3334 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:27.017948 1959 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:27.018410 3817 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.358s user 0.006s sys 0.011s
I20260927 19:38:27.018684 3584 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:27.019304 3584 raft_consensus.cc:686] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:27.020179 3584 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.021982 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:27.022871 2682 raft_consensus.cc:2354] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:27.022310 1890 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 4 to 5, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:26.994819 3745 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.995973 3741 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.026350 3741 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.024086 1890 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 4, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:27.003221 2110 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:26.960283 3772 consensus_queue.cc:1046] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260927 19:38:27.030563 3774 raft_consensus.cc:2741] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260927 19:38:26.978351 1990 raft_consensus.cc:603] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:27.032866 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:38:27.006850 1995 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:27.034580 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.004321 3825 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.004375 3829 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:38:27.009516 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 67620us
I20260927 19:38:27.006268 3432 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:26.992630 3547 raft_consensus.cc:3032] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:26.979434 3796 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:27.030783 3822 log.cc:924] Time spent T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.378s user 0.000s sys 0.008s
I20260927 19:38:26.934125 3833 consensus_queue.cc:1046] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260927 19:38:26.934657 3542 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:26.925778 3784 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.049731 3547 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:27.055441 2034 leader_election.cc:398] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:27.055202 3547 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.055874 2034 leader_election.cc:401] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 19:38:27.056339 3547 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:27.057547 3745 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.059255 3792 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.060956 3792 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:27.040819 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.069939 3784 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.070618 3825 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.073053 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.074390 3795 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.075270 2110 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:27.079823 2111 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.043279 3432 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:27.089094 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 48404.9us
I20260927 19:38:27.092538 3801 raft_consensus.cc:3032] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:27.093031 3801 raft_consensus.cc:729] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:27.093912 3801 consensus_queue.cc:260] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.095598 3801 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:27.093948 3796 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:27.096375 3796 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:27.097566 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:27.090026 2037 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.043433 3829 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:27.100117 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:27.087996 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.115496 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.116489 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.088315 3784 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:27.118785 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.122283 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.105785 3825 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.111409 3830 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.113118 3737 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.117184 3772 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:27.129681 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.124447 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.097733 3796 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.131719 3825 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.132830 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.118511 3545 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.133639 3772 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:27.137449 3824 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.123031 3779 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.123180 3799 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.132783 1987 raft_consensus.cc:1261] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:27.130352 2679 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:27.133883 3830 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.121335 3784 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:27.163065 3784 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
W20260927 19:38:27.163241 3821 log.cc:924] Time spent T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.516s user 0.001s sys 0.019s
I20260927 19:38:27.163965 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:27.164696 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:27.154881 3796 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.120362 3795 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.145998 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.126420 3741 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.149271 2142 raft_consensus.cc:2396] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:27.131608 3777 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.150719 3772 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.161787 3830 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:27.182680 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 45866.1us
I20260927 19:38:27.162096 3797 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:27.183328 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.189831 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.191527 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.194334 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:27.199134 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 75233us
I20260927 19:38:27.230516 2111 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.230755 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.193202 3830 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.235392 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.196121 3824 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.197752 3799 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.242318 3797 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260927 19:38:27.234014 3777 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:27.247246 3545 raft_consensus.cc:2741] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 19:38:27.195031 3779 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.231240 3825 raft_consensus.cc:2741] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 19:38:27.255379 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.255316 3545 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.266757 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.268181 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.269929 3745 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:27.259501 3830 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.280535 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:27.264129 3825 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.283355 1991 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:27.285537 3824 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:27.285955 3824 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:27.286319 3824 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.289655 3794 consensus_queue.cc:1046] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:27.290726 3594 raft_consensus.cc:3032] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:27.282678 3799 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260927 19:38:27.291364 3594 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:27.281791 3745 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.292466 3594 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.302165 3574 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.304050 3724 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.305754 1995 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:27.306818 3800 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.308467 3833 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:27.309338 3833 raft_consensus.cc:3032] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:27.309674 3833 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:27.310544 3741 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.310410 3833 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.310982 3741 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.311556 3833 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:27.314302 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:27.301538 3778 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.322911 2145 raft_consensus.cc:2396] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260927 19:38:27.348409 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 65667.4us
I20260927 19:38:27.349610 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.350394 2071 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:27.295020 3594 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:27.331763 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.352005 1890 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 3 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:27.281783 3779 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:27.324280 3829 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.331120 3778 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:27.365386 3829 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.367265 1989 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:27.366619 3772 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.368523 1991 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.369722 3772 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:27.371546 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.372108 1995 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.375814 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.376395 2071 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.378376 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.382763 2071 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:27.385550 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 239) took 1972 ms. Trace:
I20260927 19:38:27.390821 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.392638 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:27.317307 3432 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.391710 2142 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:27.348711 3830 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.400378 1962 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.399080 3778 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.404044 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:27.376302 3741 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.392922 1892 rpcz_store.cc:276] 0927 19:38:25.413105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:25.413247 (+ 142us) service_pool.cc:227] Handling call
0927 19:38:27.385477 (+1972230us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:25.999171 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:25.999678 (+ 507us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:25.999709 (+ 31us) write_op.cc:435] Acquired schema lock
0927 19:38:25.999727 (+ 18us) tablet.cc:660] Decoding operations
0927 19:38:26.000876 (+ 1149us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:26.000961 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:26.000991 (+ 30us) tablet.cc:683] Acquiring locks for 11 operations
0927 19:38:26.159556 (+158565us) tablet.cc:699] Row locks acquired
0927 19:38:26.162223 (+ 2667us) write_op.cc:260] PREPARE: finished
0927 19:38:26.174961 (+ 12738us) write_op.cc:270] Start()
0927 19:38:26.181514 (+ 6553us) write_op.cc:276] Timestamp: P: 1790537906174911 usec, L: 0
0927 19:38:26.184671 (+ 3157us) op_driver.cc:331] REPLICATION: starting
0927 19:38:26.218432 (+ 33761us) log.cc:842] Serialized 4408 byte log entry
0927 19:38:26.530482 (+312050us) op_driver.cc:444] REPLICATION: finished
0927 19:38:26.591800 (+ 61318us) write_op.cc:301] APPLY: starting
0927 19:38:26.591890 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:26.592155 (+ 265us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:26.592267 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:26.593628 (+ 1361us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:26.593865 (+ 237us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0927 19:38:26.593910 (+ 45us) write_op.cc:312] APPLY: finished
0927 19:38:26.594213 (+ 303us) log.cc:842] Serialized 83 byte log entry
0927 19:38:26.595002 (+ 789us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:26.595175 (+ 173us) write_op.cc:454] Released schema lock
0927 19:38:26.595224 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:26.597503 (+ 2279us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:26.597541 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12899,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":54049,"apply.run_cpu_time_us":3569,"apply.run_wall_time_us":6215,"num_ops":11,"prepare.queue_time_us":142202,"prepare.run_cpu_time_us":5891,"prepare.run_wall_time_us":219561,"replication_time_us":341056,"thread_start_us":101488,"threads_started":2}]]}
I20260927 19:38:27.403560 2064 raft_consensus.cc:1226] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:27.410838 1959 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.383559 3779 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:27.415030 2064 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260927 19:38:27.418311 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.426221 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.426924 2067 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:27.391120 3724 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.425206 2141 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:27.440557 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 78491.3us
I20260927 19:38:27.429941 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.432040 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.432273 2066 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:27.432633 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.436039 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.441771 2069 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.442055 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:27.442606 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.438900 2142 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260927 19:38:27.403126 3772 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.451546 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.403784 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:27.445027 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.429664 1961 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.472295 2068 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:27.448174 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.473315 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.457290 3830 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.457762 3745 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:27.458047 3655 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.510130 3745 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.512377 3745 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.458261 2037 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.512584 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:27.513458 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 55266.6us
I20260927 19:38:27.514217 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.514703 2685 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:27.515357 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.458789 3834 consensus_queue.cc:1046] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:27.459038 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.518141 2070 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:27.518180 3834 raft_consensus.cc:3032] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:27.518703 3834 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:27.519620 3834 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.459429 3825 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:27.521632 3825 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.523797 3432 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.524513 3825 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.460204 3747 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:27.460527 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.461052 3545 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:27.526445 3545 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.526691 3735 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:27.527249 3545 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.432060 3824 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.527989 3545 consensus_queue.cc:1046] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:27.463819 2069 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:27.466058 2063 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:27.529704 3779 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.530812 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:27.416293 3829 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:27.530300 2067 raft_consensus.cc:1261] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:27.531805 3829 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:27.532482 2138 raft_consensus.cc:2354] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:27.532719 3829 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.476436 3650 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.416471 3368 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:27.477151 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.478889 1991 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:27.479465 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:27.534273 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 57217.1us
I20260927 19:38:27.480090 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.534986 1961 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.502197 3802 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.536010 3802 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.537050 3334 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.537696 3334 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.537801 3728 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:27.539000 3802 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:27.506892 3845 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260927 19:38:27.540501 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.482102 3772 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.417824 3778 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.486513 2682 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:27.542536 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.476428 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:27.543421 3778 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:27.544076 3839 log.cc:924] Time spent T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260927 19:38:27.446182 3799 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:27.530848 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 70387.6us
I20260927 19:38:27.545208 3799 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.546164 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.547216 3799 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.539208 3728 raft_consensus.cc:979] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:27.477038 3542 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.548264 3728 raft_consensus.cc:1067] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:27.457381 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.549037 2136 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:27.542569 3772 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:27.502493 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.552289 1989 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260927 19:38:27.539841 1994 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:27.562449 3828 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:27.565003 1987 raft_consensus.cc:2379] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:27.565026 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:27.550680 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:27.547757 3799 raft_consensus.cc:3032] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:27.577677 3799 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:27.529724 3824 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.578694 3799 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.580189 3829 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.580960 2682 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
W20260927 19:38:27.586658 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47677.3us
I20260927 19:38:27.587153 3838 raft_consensus.cc:2741] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 19:38:27.587764 1961 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.587901 1959 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.588953 1961 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.572894 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:27.596089 2686 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:27.590510 3724 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.568858 1993 raft_consensus.cc:3032] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:27.600220 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.597113 1993 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:27.603833 1961 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.599020 1995 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:27.607511 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 91050.1us
I20260927 19:38:27.609607 2037 leader_election.cc:398] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:27.610176 2037 leader_election.cc:401] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260927 19:38:27.611788 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.591686 3784 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.617154 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.618551 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.619029 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.619405 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.619621 2063 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:27.619903 2067 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:27.620080 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.621203 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.622208 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.625587 2070 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.626230 1961 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.628213 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.630008 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.630584 2069 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:27.601886 3584 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:27.636713 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.639634 2069 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:27.640897 2139 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:27.629330 2063 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:27.639320 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.638950 3650 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.645084 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.640266 3794 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.645476 3813 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:27.646279 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.606340 3741 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:27.613307 3334 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.645815 2141 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:27.645488 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.649271 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.614181 1993 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.655635 1993 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:27.650098 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.633515 3834 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.651317 1995 raft_consensus.cc:2379] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:27.653573 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.630208 3824 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:27.650169 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.656234 2069 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:27.638337 3745 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.662654 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.663399 2069 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:27.664258 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.652082 2067 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:27.617304 3779 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.660699 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.661875 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.671762 2066 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:27.672021 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.645573 3584 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:27.665942 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.666131 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.666658 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:27.677665 2141 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.677839 2070 raft_consensus.cc:1261] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:27.661832 3824 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:27.709573 3824 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.711596 2137 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:27.711393 3584 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.653378 3819 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260927 19:38:27.672829 1994 raft_consensus.cc:2379] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:27.719668 2141 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:27.675482 2068 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:27.663563 3830 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.653602 3594 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260927 19:38:27.654098 3853 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:27.679132 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 53405.1us
I20260927 19:38:27.705332 3334 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:27.672880 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:27.713461 3794 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.748698 3650 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:27.671062 3745 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.731159 3829 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:27.764097 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 91284.6us
I20260927 19:38:27.732093 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:27.786288 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54296.6us
I20260927 19:38:27.774047 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.787554 2144 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:27.790490 2110 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:27.792089 2110 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.795204 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:27.722797 1988 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:27.795993 3741 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:27.808171 2110 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.814106 1961 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.723230 3830 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.735190 3747 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:27.772119 3853 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:27.839022 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.839928 3432 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:27.783742 3334 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:27.859964 3432 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:27.860369 3432 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.860878 3334 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.743006 2065 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:27.849345 3542 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:27.850519 3737 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:27.868171 3799 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.868887 1988 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:27.851949 3728 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
W20260927 19:38:27.870889 1935 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260927 19:38:27.807081 3840 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.854516 3747 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.873483 3747 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.772686 3784 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:27.782917 3794 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.876675 3794 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:27.836109 3797 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.877427 3797 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.878995 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.879386 3797 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:27.880247 2136 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:27.815517 3778 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:27.881098 3778 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:27.881524 3778 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:27.857887 3852 consensus_queue.cc:1046] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260927 19:38:27.871872 3824 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.887691 3824 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:27.844974 1993 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:27.769899 3772 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.869727 3741 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:27.851037 3745 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:27.891120 3545 raft_consensus.cc:2741] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260927 19:38:27.855551 3792 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:27.899832 2140 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:27.901132 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:27.905910 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:27.948789 2139 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.907755 2654 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.919917 3368 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:27.925771 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 86860.3us
W20260927 19:38:27.950996 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 155889us
I20260927 19:38:27.951279 1961 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.927969 3650 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.931653 3432 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.932192 3779 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:27.809448 3800 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:27.952119 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.954010 1961 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:27.954779 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 47138.9us
I20260927 19:38:27.963456 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.966157 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.966928 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:27.973855 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.974766 2069 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:27.975571 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.978096 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:27.979156 2070 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:27.979403 2111 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:38:27.978097 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:27.981961 2068 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:27.983126 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.983417 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.983736 1991 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:27.976202 2069 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:27.987715 2111 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.989077 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.989885 1995 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 2: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:27.990424 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.991218 1993 raft_consensus.cc:2379] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:27.988675 2139 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:27.993510 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:27.994074 2068 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:27.994410 2068 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:27.995288 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:27.997229 1962 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:27.999940 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:27.943809 3772 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.970458 3824 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.967228 3800 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:28.001003 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:28.001052 2140 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:27.986850 3799 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:27.969776 3545 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:27.964802 3840 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20260927 19:38:28.008049 2063 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:27.970430 3857 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:27.980376 3542 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:28.014876 3724 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:28.002555 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.964840 2679 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 1.
I20260927 19:38:27.986721 3574 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.002257 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:28.050776 2069 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:28.052732 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:27.947882 3792 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:27.996752 3745 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:28.012789 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:27.965312 3368 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.000818 3794 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:27.989012 3737 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20260927 19:38:28.012787 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:28.041710 1996 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:28.044703 3432 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.020287 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:28.053339 1991 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.055186 1993 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:28.001050 3797 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:28.023597 3800 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:28.059202 2034 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:28.061983 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:28.012985 2068 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.067329 2111 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:28.071179 2064 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.072645 2070 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:28.050840 3799 raft_consensus.cc:686] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:28.057117 3834 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:28.056988 3745 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:28.062430 3825 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:28.023348 3830 consensus_queue.cc:1046] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 19:38:28.064131 3724 raft_consensus.cc:686] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:28.022477 3778 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:28.088588 2068 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:28.064628 3852 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:28.064939 3794 raft_consensus.cc:686] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:28.069690 3542 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:28.073240 3797 raft_consensus.cc:686] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:28.108323 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:28.056771 3574 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.082280 3779 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.089449 3802 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:28.088181 3745 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:28.119948 2068 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:28.124080 3779 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.123188 3802 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.111640 3829 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.155678 3797 consensus_queue.cc:237] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:28.124537 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 120426us
I20260927 19:38:28.153496 3724 consensus_queue.cc:237] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.151671 3794 consensus_queue.cc:237] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.149844 3799 consensus_queue.cc:237] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.144810 3542 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.170341 2068 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 19:38:28.168676 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 45394.6us
I20260927 19:38:28.176388 3772 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.177611 3368 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:28.123586 3777 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:28.179108 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:28.174309 3802 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:28.202586 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:28.203310 1995 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:28.198566 2144 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:28.206380 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:28.206988 1995 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:28.207700 2111 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:28.208647 2111 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:28.210461 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 215211us
I20260927 19:38:28.213742 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:28.215406 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:28.216105 1995 raft_consensus.cc:3032] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:28.216456 1995 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:28.117805 3852 raft_consensus.cc:686] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:28.224159 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:28.225109 1992 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:28.194978 3862 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.165s user 0.000s sys 0.002s
I20260927 19:38:28.227577 1995 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.241590 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:28.242414 2141 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.244670 1995 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:28.252990 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 49890.2us
I20260927 19:38:28.254470 2111 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:28.227638 3833 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:28.262440 2110 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:28.183604 3829 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.228684 3772 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:28.075964 3864 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:28.264947 3864 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.265841 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:28.267660 1991 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:28.233867 3801 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:28.261132 3432 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.275243 1995 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:28.269745 2069 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:28.321144 2139 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260927 19:38:28.322252 2141 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:28.322646 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:28.323402 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:28.321651 2139 pending_rounds.cc:76] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 1
I20260927 19:38:28.326468 2139 pending_rounds.cc:97] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:38:28.326762 2139 raft_consensus.cc:2866] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:28.327798 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.328158 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:28.328953 2063 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:28.328919 2069 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.330935 3852 consensus_queue.cc:237] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.266448 3772 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.339088 3772 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.351732 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:28.353147 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.262593 3792 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.268342 3833 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.270123 3574 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:28.370105 3868 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260927 19:38:28.327849 3432 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.372486 3334 raft_consensus.cc:2741] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260927 19:38:28.276278 3875 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.136s
I20260927 19:38:28.266242 3864 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:28.293088 3801 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.354034 2063 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:28.257704 3650 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.086s
W20260927 19:38:28.392426 3863 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
W20260927 19:38:28.420630 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 158658us
I20260927 19:38:28.380406 1988 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:28.380606 3796 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:28.415071 3882 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:28.429790 2069 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:28.426506 2111 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:28.409744 3833 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:28.476042 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 49608.9us
I20260927 19:38:28.476827 2111 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:38:28.481537 3797 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:28.482072 3797 raft_consensus.cc:686] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:28.482846 3797 consensus_queue.cc:237] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.484218 2064 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:28.422317 3801 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:28.486654 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 224321us
W20260927 19:38:28.487114 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 200798us
I20260927 19:38:28.426558 3796 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:38:28.485905 3878 log.cc:924] Time spent T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260927 19:38:28.473134 3778 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:28.492799 3574 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:28.496109 3796 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:28.500269 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 176926us
I20260927 19:38:28.499517 3882 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.516909 3778 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.516666 3574 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:38:28.569613 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 92845.1us
I20260927 19:38:28.521624 3885 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:28.626219 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 101825us
W20260927 19:38:28.637756 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 113631us
I20260927 19:38:28.592718 3574 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.650875 2686 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:38:28.655300 3888 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:28.648873 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:28.619899 3829 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:28.657223 2136 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.585s user 0.008s sys 0.007s
I20260927 19:38:28.657848 2680 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:28.523329 3882 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.711975 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 221) took 1131 ms. Trace:
W20260927 19:38:28.704941 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 58353.8us
I20260927 19:38:28.621385 3778 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.723732 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:28.724535 1995 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
W20260927 19:38:28.731662 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 409065us
I20260927 19:38:28.733063 3545 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:28.733655 3545 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
W20260927 19:38:28.734789 3877 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.246s user 0.001s sys 0.000s
I20260927 19:38:28.734640 3545 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.742396 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.740172 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.745926 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.742229 2063 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:28.741834 2111 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:28.777626 2064 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:28.777830 2068 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:28.778887 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:28.780615 2064 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:28.781286 2071 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 1.
I20260927 19:38:28.704234 2136 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
W20260927 19:38:28.749136 3800 consensus_meta.cc:287] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.681s user 0.013s sys 0.000s
I20260927 19:38:28.640288 3745 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.804502 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:38:28.632599 3885 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:28.815366 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 79944.9us
I20260927 19:38:28.813927 3779 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:28.786563 3432 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.766734 3792 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:28.808382 2064 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:28.793551 3797 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260927 19:38:28.829793 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 85227us
I20260927 19:38:28.801795 3745 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:28.833873 3879 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:28.834371 3879 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:28.834800 3879 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:28.835049 1961 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:28.839965 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:28.840260 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:28.840687 1994 raft_consensus.cc:2379] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
W20260927 19:38:28.848965 3892 log.cc:924] Time spent T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20260927 19:38:28.850107 3796 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.852409 3796 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.799964 3800 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.854157 3778 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.854425 3800 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.719684 1987 rpcz_store.cc:276] 0927 19:38:27.580813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:27.595725 (+ 14912us) service_pool.cc:227] Handling call
0927 19:38:27.639433 (+ 43708us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:27.713843 (+ 74410us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 19:38:27.737869 (+ 24026us) log.cc:842] Serialized 14 byte log entry
0927 19:38:27.964287 (+226418us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:28.027129 (+ 62842us) log.cc:842] Serialized 26 byte log entry
0927 19:38:28.050586 (+ 23457us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:28.057304 (+ 6718us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:28.711219 (+653915us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b5400285b20. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138f2730 00007f7913965b42 00007f7913962f2e 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:38:28.711363 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:28.711445 (+ 82us) raft_consensus.cc:1671] finished
0927 19:38:28.711611 (+ 166us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:28.711847 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23020,"prepare.queue_time_us":32564,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":24850,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":11323,"spinlock_wait_cycles":34023296,"thread_start_us":204497,"threads_started":1,"wal-append.queue_time_us":285376}
W20260927 19:38:28.858966 3880 log.cc:924] Time spent T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.286s user 0.004s sys 0.006s
I20260927 19:38:28.859191 3885 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:28.860281 3894 log.cc:924] Time spent T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260927 19:38:28.836092 3797 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.861344 3834 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:28.877396 3834 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:28.877991 1994 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:28.878161 3876 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:28.878783 3876 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:28.881456 3885 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:28.882164 3882 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.844447 3829 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:28.883463 3829 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:28.883874 3829 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:28.884085 3802 consensus_meta.cc:287] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.653s user 0.007s sys 0.007s
I20260927 19:38:28.884481 3802 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:28.888103 3899 log.cc:924] Time spent T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
W20260927 19:38:28.888691 3893 log.cc:924] Time spent T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260927 19:38:28.888103 3833 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:28.890174 3898 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260927 19:38:28.838330 3792 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:28.890626 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.892051 3792 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:28.880892 3876 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:28.874837 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.894702 2062 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:28.841022 1989 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:28.844130 3828 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:28.923153 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.874358 3797 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:28.937076 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:28.927170 2070 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:28.933784 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:28.938881 2110 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:28.935457 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:28.940274 2062 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:28.939818 3884 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260927 19:38:28.878094 3866 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:28.892057 3833 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.949954 3905 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:28.951686 2138 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:28.988583 3900 log.cc:924] Time spent T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260927 19:38:28.990445 3870 log.cc:924] Time spent T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.757s user 0.003s sys 0.000s
I20260927 19:38:28.990698 1989 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:28.991638 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1317) took 1019 ms. Trace:
I20260927 19:38:28.952026 3802 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:28.954198 2141 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:28.954735 3884 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:28.954727 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:28.955008 2680 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:28.945959 3824 consensus_queue.cc:1046] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260927 19:38:28.932742 3828 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:28.962493 3906 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:28.963380 3801 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.970057 3853 raft_consensus.cc:3032] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:28.981987 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.024083 2066 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 3
I20260927 19:38:28.985809 3799 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260927 19:38:28.992043 2067 rpcz_store.cc:276] 0927 19:38:27.971968 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:27.978152 (+ 6184us) service_pool.cc:227] Handling call
0927 19:38:27.978353 (+ 201us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:27.999791 (+ 21438us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:27.999828 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:28.000237 (+ 409us) log.cc:842] Serialized 26 byte log entry
0927 19:38:28.110568 (+110331us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:28.110611 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:28.110677 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:28.847770 (+737093us) spinlock_profiling.cc:243] Waited 47.5 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:28.991267 (+143497us) raft_consensus.cc:1671] finished
0927 19:38:28.991292 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:28.991575 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11553,"raft.queue_time_us":48769,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":25285,"spinlock_wait_cycles":109203712,"thread_start_us":78198,"threads_started":1,"wal-append.queue_time_us":145448}
I20260927 19:38:28.995836 2145 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:28.907408 3885 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:28.988611 1988 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260927 19:38:29.019515 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 184533us
I20260927 19:38:29.018700 2686 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260927 19:38:29.035187 2139 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:29.042531 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:29.042866 2141 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:29.011603 3830 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:29.045712 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 213613us
I20260927 19:38:29.051712 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.059320 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:29.028575 3772 consensus_meta.cc:287] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.689s user 0.003s sys 0.012s
I20260927 19:38:29.070487 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.027200 2065 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 3
W20260927 19:38:29.081919 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52357.3us
I20260927 19:38:29.026031 3853 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:29.089118 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.090330 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.090952 2064 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.033730 3886 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:29.092898 3913 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:28.988634 3905 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.045437 3724 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260927 19:38:29.033732 3828 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.023437 3884 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.033919 3879 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.032037 3824 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:29.096036 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48923.2us
I20260927 19:38:29.100338 2067 raft_consensus.cc:1261] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:29.100251 2143 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:29.114270 2068 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:29.105649 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.100550 2066 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:28.986979 3866 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.128646 2066 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:29.129420 2110 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.093245 3886 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.095006 3876 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.134469 2064 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.093639 3888 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.135814 2071 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:29.097988 2679 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:38:29.136828 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.031817 3857 consensus_queue.cc:1046] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20260927 19:38:29.107635 3907 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:29.062071 3745 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260927 19:38:29.031641 3887 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260927 19:38:29.164000 3857 consensus_queue.cc:1225] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer 69f09866d0a1477585f05b2f82ff4c67 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260927 19:38:29.164950 3887 raft_consensus.cc:3032] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:29.165318 3887 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:29.143730 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.166164 3887 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.143319 2110 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.144246 1994 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:29.061969 3830 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.096349 3650 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260927 19:38:29.029956 3906 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.175534 2137 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:29.121968 3824 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.176753 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.177026 3906 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.087917 3772 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.177340 2068 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.142716 3876 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.178284 3824 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.134370 3882 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260927 19:38:29.153854 2141 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:29.121589 3913 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.181262 3913 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.181720 3866 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.182192 3737 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:29.182731 2138 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:38:29.101890 3853 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:29.186165 3574 consensus_meta.cc:287] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.525s user 0.009s sys 0.000s
I20260927 19:38:29.186432 3853 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.143740 3888 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.188831 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.189002 3888 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.157650 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.197201 1992 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:29.137569 3886 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.162036 3801 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.162241 3907 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:29.179462 3876 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.192103 1995 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:29.155002 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.209007 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.194233 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:29.194365 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.177433 3432 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:29.218951 2067 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
W20260927 19:38:29.218258 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 386944us
I20260927 19:38:29.224213 2068 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.167657 3794 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:29.166877 3869 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260927 19:38:29.231384 2108 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:29.221318 2070 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:29.226780 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.229699 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.229015 2653 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:29.232913 2068 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.186638 3574 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.236810 3574 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.196537 3871 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:29.237602 2142 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:29.238284 3871 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.240372 3871 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.240530 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.179479 3772 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.189769 3887 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:29.248068 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.250957 2068 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
W20260927 19:38:29.231073 3868 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 19:38:29.256814 2064 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:29.254251 2145 raft_consensus.cc:2379] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:29.260898 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.261545 2068 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:29.218855 3737 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 19:38:29.263538 3737 consensus_queue.cc:1225] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer 69f09866d0a1477585f05b2f82ff4c67 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260927 19:38:29.264159 3796 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:29.264817 3924 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:29.265622 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.265388 3924 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.266415 2143 raft_consensus.cc:2396] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:29.268575 3924 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.270002 3912 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260927 19:38:29.270385 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:29.270851 3432 raft_consensus.cc:686] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.271838 3432 consensus_queue.cc:237] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.270961 2143 raft_consensus.cc:595] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:29.272377 3918 log.cc:924] Time spent T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260927 19:38:29.196774 3368 consensus_queue.cc:1046] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260927 19:38:29.243341 3906 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:29.242192 1988 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.281728 3794 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.245281 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:29.291954 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 63043.6us
I20260927 19:38:29.286702 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.282662 3885 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.290935 1995 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:29.253110 3724 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.299342 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.316517 2141 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
W20260927 19:38:29.300999 3864 consensus_meta.cc:287] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.875s user 0.011s sys 0.000s
I20260927 19:38:29.300068 2145 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
W20260927 19:38:29.302850 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 73232.8us
I20260927 19:38:29.305486 3796 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.305382 3797 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.306180 3772 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
W20260927 19:38:29.307406 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 59736.6us
I20260927 19:38:29.322422 2111 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:29.259508 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.325376 2137 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:29.309612 3828 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.342981 3828 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.309990 2138 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:29.311558 3829 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.312377 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.348250 2686 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:29.319329 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.349321 2034 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:29.349660 1995 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:29.319530 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.319983 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:29.299372 3741 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:29.324030 2110 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.328487 2111 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.328879 1992 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:29.330159 3794 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.330672 3908 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:29.316951 3747 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:29.334529 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.338796 3905 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.332845 3834 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.284081 3830 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.341120 1996 raft_consensus.cc:1261] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:29.342446 3879 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.263181 3888 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.009s
I20260927 19:38:29.377664 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.350477 2145 raft_consensus.cc:595] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:29.352954 2144 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:29.352965 3828 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.351538 3741 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:29.346912 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.353390 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.381574 1992 raft_consensus.cc:2379] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:29.362550 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.383400 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.341249 3907 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.355895 3796 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.382536 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.339672 2065 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:29.378499 2145 raft_consensus.cc:603] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:29.361070 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.371232 3792 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:29.364034 3772 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.364228 3747 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.371403 3833 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:29.414000 3833 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.414438 3833 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.378289 1996 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.337021 3886 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.385833 2688 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.371694 3884 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.387507 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:29.387748 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 197157us
I20260927 19:38:29.388123 1995 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:29.321214 3724 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.386085 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.406667 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:29.189706 3779 consensus_queue.cc:1159] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "69f09866d0a1477585f05b2f82ff4c67" responder_term: 4 status { last_received { term: 4 index: 2 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 19:38:29.408895 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.428799 2137 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:29.429029 3829 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.429260 2138 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:29.411588 3772 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.327800 3864 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.417927 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.417841 3747 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.451253 3864 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.440295 1962 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.397171 3741 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.452858 3741 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.417701 3792 raft_consensus.cc:686] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.454469 3792 consensus_queue.cc:237] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.427074 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.443527 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:29.441111 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.448589 1996 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.451164 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.442094 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.449991 3797 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.414438 3834 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.459162 1962 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:29.465492 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.467417 3888 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:29.433574 3432 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:29.438210 3655 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:29.471993 2108 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.452831 3368 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:29.471248 3831 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:29.507019 3831 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:29.507427 3831 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:29.518538 3901 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:29.519079 3901 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.519454 3901 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:29.498513 3368 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.520283 3368 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:29.498296 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.495779 3655 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:29.514811 1989 raft_consensus.cc:2396] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:29.468940 3833 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:29.544749 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 288060us
I20260927 19:38:29.522169 3728 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.525341 3778 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.547334 2111 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.503042 3885 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:29.498085 3432 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.553428 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.555570 1996 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.541216 3655 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:29.559144 3885 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.568059 3885 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.571938 2143 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:29.569321 3737 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.558135 3778 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.548704 2137 raft_consensus.cc:595] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:29.574693 3737 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.578415 3778 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.582954 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.580878 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.586473 3776 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.586747 3872 raft_consensus.cc:979] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:29.586510 3724 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.584396 3542 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260927 19:38:29.556663 3796 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:29.584815 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.590662 2680 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:29.594228 3737 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.598623 3833 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.625511 1993 raft_consensus.cc:1261] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:29.576548 3594 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260927 19:38:29.626734 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.554378 3728 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.627641 1993 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:29.628953 3728 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.610407 3776 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.629981 1993 raft_consensus.cc:1261] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260927 19:38:29.630870 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 154) took 3693 ms. Trace:
I20260927 19:38:29.581627 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:29.623883 3925 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.175s user 0.000s sys 0.002s
I20260927 19:38:29.624807 3796 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.630934 3776 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.571128 3794 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:29.636852 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.638886 2108 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.645164 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.646011 1993 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.649091 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:29.651597 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 77934.2us
I20260927 19:38:29.642163 3801 raft_consensus.cc:1067] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Signalling peer 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:38:29.653937 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.661257 2143 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:29.654664 1996 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.663679 3796 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.649338 1987 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:29.664428 2654 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.672654 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.674001 2145 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:29.675988 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.676234 3432 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.676541 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.647866 1988 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.678452 3838 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:29.678982 3838 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.670893 3594 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:29.646975 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:29.684574 2138 raft_consensus.cc:595] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:29.685065 2138 raft_consensus.cc:603] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:29.676681 2141 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:29.688730 1992 raft_consensus.cc:2396] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:29.689164 3655 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.695963 3890 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:29.712349 3890 raft_consensus.cc:686] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:29.713399 3890 consensus_queue.cc:237] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.682682 3838 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.684859 3368 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.719352 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.631346 1890 rpcz_store.cc:276] 0927 19:38:25.938048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:25.938317 (+ 269us) service_pool.cc:227] Handling call
0927 19:38:29.630815 (+3692498us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:27.722895 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:27.723425 (+ 530us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:27.723456 (+ 31us) write_op.cc:435] Acquired schema lock
0927 19:38:27.723471 (+ 15us) tablet.cc:660] Decoding operations
0927 19:38:27.724010 (+ 539us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:27.724082 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:27.724106 (+ 24us) tablet.cc:683] Acquiring locks for 3 operations
0927 19:38:27.724423 (+ 317us) tablet.cc:699] Row locks acquired
0927 19:38:27.724440 (+ 17us) write_op.cc:260] PREPARE: finished
0927 19:38:27.724574 (+ 134us) write_op.cc:270] Start()
0927 19:38:27.724671 (+ 97us) write_op.cc:276] Timestamp: P: 1790537907724535 usec, L: 0
0927 19:38:27.724692 (+ 21us) op_driver.cc:331] REPLICATION: starting
0927 19:38:27.725263 (+ 571us) log.cc:842] Serialized 1302 byte log entry
0927 19:38:28.790683 (+1065420us) op_driver.cc:444] REPLICATION: finished
0927 19:38:29.149178 (+358495us) write_op.cc:301] APPLY: starting
0927 19:38:29.173187 (+ 24009us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:29.207388 (+ 34201us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:29.232926 (+ 25538us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:29.233462 (+ 536us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:29.233623 (+ 161us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0927 19:38:29.233674 (+ 51us) write_op.cc:312] APPLY: finished
0927 19:38:29.233874 (+ 200us) log.cc:842] Serialized 35 byte log entry
0927 19:38:29.234031 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:29.234116 (+ 85us) write_op.cc:454] Released schema lock
0927 19:38:29.234151 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:29.234411 (+ 260us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:29.234440 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":566969,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":330621,"apply.run_cpu_time_us":2427,"apply.run_wall_time_us":94778,"num_ops":3,"prepare.queue_time_us":270733,"prepare.run_cpu_time_us":141244,"prepare.run_wall_time_us":307386,"raft.queue_time_us":220141,"raft.run_cpu_time_us":58576,"raft.run_wall_time_us":289265,"replication_time_us":1059278,"thread_start_us":698607,"threads_started":4,"wal-append.queue_time_us":367518}]]}
I20260927 19:38:29.709725 3594 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.720561 3594 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:29.721468 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.710012 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.710273 3728 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:29.719944 2142 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c802eef7b2904ec984925e3999cff361"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:29.709259 3801 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:29.736083 3728 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:29.736555 3776 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:29.737057 3728 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.724606 1991 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:29.724960 3906 raft_consensus.cc:686] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
W20260927 19:38:29.738757 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 419297us
I20260927 19:38:29.654125 3794 raft_consensus.cc:686] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.740257 1993 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:29.741225 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.741730 2653 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:29.741987 1993 raft_consensus.cc:2354] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4.
I20260927 19:38:29.742105 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.736310 3831 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:29.738713 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 389428us
I20260927 19:38:29.743921 3852 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:29.676573 3747 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.709488 3778 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:29.743943 3368 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.745426 1988 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:38:29.705951 3886 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:29.758106 2686 raft_consensus.cc:2354] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4.
I20260927 19:38:29.758201 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.753890 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.759517 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.786742 2111 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.750590 3905 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:29.791188 2145 raft_consensus.cc:2354] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4.
I20260927 19:38:29.791281 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.748728 3834 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:29.748589 3852 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.809938 3852 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:29.811053 3831 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.798446 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:29.750496 3778 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.813581 3778 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.750727 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.754777 2137 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:29.750321 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:29.814855 3834 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:29.815788 3834 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.816944 2138 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:29.817523 3745 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:29.818040 3745 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.819237 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.820011 3745 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.821661 3884 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.748032 3906 consensus_queue.cc:237] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.821770 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.822144 3884 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.824234 3884 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.824731 3884 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:29.826261 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.750602 3747 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.827453 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.826759 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.828531 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.829274 2064 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:29.801434 3655 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.801326 3794 consensus_queue.cc:237] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.802990 3866 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:29.804347 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.804235 3901 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.808563 2111 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:29.809191 3574 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:29.833559 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.841241 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:29.841385 2070 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:29.842489 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:29.842581 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.795702 3368 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:29.848265 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.795770 1987 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:60286
I20260927 19:38:29.793466 3905 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:29.820044 1989 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
W20260927 19:38:29.849450 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45186.5us
I20260927 19:38:29.826748 2680 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:29.825173 3884 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.838366 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:29.849697 3886 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:29.857973 2064 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:29.876231 3886 raft_consensus.cc:686] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.879922 3886 consensus_queue.cc:237] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.884195 3864 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:29.885848 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 47574.7us
I20260927 19:38:29.887085 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:29.888018 3864 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:29.889568 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:29.851326 3833 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:29.854409 3888 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:29.852720 3776 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:29.859135 3905 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:29.853818 3574 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:29.855176 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.870229 1987 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:29.900295 3864 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:29.902593 2037 leader_election.cc:398] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4.
I20260927 19:38:29.869683 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.847464 3866 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.911777 2037 leader_election.cc:401] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:29.901530 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:29.924196 3834 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.925464 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.926498 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.927299 2137 raft_consensus.cc:2354] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:29.928397 2654 leader_election.cc:398] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:29.854808 3368 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.928681 2654 leader_election.cc:401] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 19:38:29.914098 3574 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:29.906646 1987 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.919550 3866 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.919973 1989 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:29.921008 2145 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:29.913511 2138 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.897941 3888 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:29.930250 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:29.947290 2143 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:29.933539 3818 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:29.933833 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:29.950143 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.939903 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:29.919553 3884 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.958679 3655 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 19:38:29.960062 3933 log.cc:924] Time spent T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260927 19:38:29.975629 3594 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.977504 2141 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:29.978988 1893 catalog_manager.cc:5495] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:29.983342 2688 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 19:38:29.959899 3924 raft_consensus.cc:3032] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:29.986781 3924 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:29.960897 3901 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:29.963092 2108 leader_election.cc:398] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4.
I20260927 19:38:29.989995 2108 leader_election.cc:401] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:29.964080 3852 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:29.949194 3888 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.951788 3866 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260927 19:38:29.952659 2065 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:29.906801 3745 consensus_queue.cc:1046] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260927 19:38:29.936483 3779 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:29.985796 3368 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:29.988903 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:29.992682 3885 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:29.945531 3917 consensus_queue.cc:1046] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260927 19:38:29.957895 1990 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:29.990978 1893 catalog_manager.cc:5495] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:30.020373 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:30.037390 1989 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:30.002244 3852 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:30.009598 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:29.954420 3829 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260927 19:38:30.012236 2064 raft_consensus.cc:1261] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:30.013849 2069 raft_consensus.cc:1261] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260927 19:38:30.044262 2069 pending_rounds.cc:76] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 1
I20260927 19:38:30.044569 2069 pending_rounds.cc:97] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260927 19:38:30.044878 2069 raft_consensus.cc:2866] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:30.017787 2143 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:30.030292 2140 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:30.031268 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:29.988555 3802 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260927 19:38:30.039862 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:30.105223 1962 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:30.106483 1962 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:30.113672 2067 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:30.144559 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:30.153589 2140 raft_consensus.cc:2379] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:30.125200 3829 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:30.130095 2679 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:30.130596 3368 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:30.179991 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:30.182054 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:30.182245 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:30.182907 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:30.211625 2069 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:30.216380 2140 raft_consensus.cc:1226] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:30.217826 2071 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:30.151463 3834 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:30.221832 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:30.226294 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:30.234982 2071 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 19:38:30.244061 2139 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.011s sys 0.007s
W20260927 19:38:30.249786 2139 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.832s user 0.011s sys 0.008s
I20260927 19:38:30.249532 2067 raft_consensus.cc:1261] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 19:38:30.244187 3735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.002s sys 0.016s
W20260927 19:38:30.252574 3735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.002s sys 0.016s
I20260927 19:38:30.253895 2067 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 19:38:30.246905 3594 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:30.247732 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:30.248303 2063 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:30.261404 2067 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:30.213806 3829 raft_consensus.cc:686] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:30.209518 3368 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:30.307008 2140 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:30.321063 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:30.324509 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 60012.5us
I20260927 19:38:30.390327 2069 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:30.441318 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:30.785778 2068 raft_consensus.cc:2354] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3.
I20260927 19:38:30.305379 3852 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:30.232420 3813 consensus_queue.cc:1046] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.189s
I20260927 19:38:30.327952 3830 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:30.435350 3879 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:30.413486 3776 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:30.467732 1996 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.019s sys 0.037s
I20260927 19:38:30.427191 3824 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:30.474782 3886 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.018s sys 0.037s
I20260927 19:38:30.468291 3885 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:30.530223 2070 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.025s sys 0.027s
I20260927 19:38:30.490147 3930 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:30.543558 2143 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260927 19:38:30.530722 1987 consensus_meta.cc:287] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.592s user 0.011s sys 0.000s
W20260927 19:38:30.533056 3890 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s user 0.029s sys 0.045s
W20260927 19:38:30.857466 3890 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.141s user 0.030s sys 0.045s
I20260927 19:38:30.821175 3830 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:30.540517 3906 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.026s sys 0.060s
W20260927 19:38:30.858177 3906 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s user 0.026s sys 0.060s
I20260927 19:38:30.858795 3906 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:30.859468 3830 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:30.859332 3906 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:30.860100 3830 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:30.860617 3830 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:30.540403 2062 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.018s sys 0.026s
W20260927 19:38:30.699072 1989 consensus_meta.cc:287] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.661s user 0.009s sys 0.009s
I20260927 19:38:30.861624 1989 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:30.548247 3869 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:30.866640 1992 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 19:38:30.868300 1992 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:30.601681 3866 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:30.869990 3940 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.495s user 0.001s sys 0.000s
I20260927 19:38:30.869743 3866 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:30.869521 3829 consensus_queue.cc:237] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:30.871337 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 1933) took 1444 ms. Trace:
I20260927 19:38:30.871871 3866 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:30.872496 3866 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:30.861717 1987 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:30.831100 3886 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.947s user 0.019s sys 0.037s
I20260927 19:38:30.696866 3745 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:30.778947 3368 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:30.821173 1996 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s user 0.019s sys 0.037s
W20260927 19:38:30.821381 3871 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.043s sys 0.114s
W20260927 19:38:30.849006 2070 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.027s sys 0.032s
I20260927 19:38:30.830178 3824 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:30.866076 2062 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s user 0.020s sys 0.032s
I20260927 19:38:30.697383 3907 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:30.874747 1989 raft_consensus.cc:1261] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:30.889930 1992 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20260927 19:38:30.910686 1992 pending_rounds.cc:76] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Aborting all ops after (but not including) 1
W20260927 19:38:30.784574 3772 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.322s user 0.066s sys 0.087s
I20260927 19:38:30.866466 3869 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:30.852761 3930 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:30.911058 1992 pending_rounds.cc:97] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260927 19:38:30.923316 1992 raft_consensus.cc:2866] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 19:38:30.900270 3871 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.947s user 0.045s sys 0.114s
W20260927 19:38:30.899099 3888 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.896s user 0.011s sys 0.007s
I20260927 19:38:30.873049 3866 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:30.920390 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:30.873924 3334 consensus_queue.cc:1046] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260927 19:38:30.826099 3879 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:30.892570 3724 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:38:30.903810 3905 consensus_meta.cc:287] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.998s user 0.006s sys 0.005s
I20260927 19:38:30.831787 3776 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:30.934374 2653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.034s sys 0.026s
W20260927 19:38:30.953651 2653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.034s sys 0.026s
I20260927 19:38:30.890269 3885 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:30.918972 3772 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.464s user 0.067s sys 0.087s
W20260927 19:38:30.938701 2139 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s user 0.020s sys 0.040s
I20260927 19:38:30.903633 3906 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:30.912931 1988 raft_consensus.cc:1226] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
W20260927 19:38:30.942782 2688 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.019s sys 0.042s
W20260927 19:38:30.952988 1995 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.032s sys 0.055s
W20260927 19:38:31.014484 2688 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s user 0.019s sys 0.042s
W20260927 19:38:31.014601 1995 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.027s user 0.033s sys 0.055s
I20260927 19:38:30.821722 3852 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:30.972123 2139 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s user 0.020s sys 0.040s
I20260927 19:38:30.898327 3745 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.015806 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 241) took 1545 ms. Trace:
I20260927 19:38:31.019907 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 239) took 1639 ms. Trace:
I20260927 19:38:31.019376 1995 rpcz_store.cc:276] 0927 19:38:29.470276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.470608 (+ 332us) service_pool.cc:227] Handling call
0927 19:38:29.471214 (+ 606us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:29.471378 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 19:38:29.474071 (+ 2693us) log.cc:842] Serialized 14 byte log entry
0927 19:38:29.987397 (+513326us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b180021d9c0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e957b 00007f791396c88a
0927 19:38:31.015261 (+1027864us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:31.015322 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:31.015367 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.015453 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.015735 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1448,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"spinlock_wait_cycles":49171712,"thread_start_us":1027616,"threads_started":1}
I20260927 19:38:31.022450 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:31.023234 2068 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:31.026322 1994 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.200s user 0.046s sys 0.123s
W20260927 19:38:31.026716 1994 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.200s user 0.046s sys 0.123s
W20260927 19:38:31.016884 3794 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.112s user 0.030s sys 0.074s
I20260927 19:38:31.020596 2139 rpcz_store.cc:276] 0927 19:38:29.380616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.386701 (+ 6085us) service_pool.cc:227] Handling call
0927 19:38:29.386957 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:29.388323 (+ 1366us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 19:38:30.250172 (+861849us) log.cc:842] Serialized 14 byte log entry
0927 19:38:31.019455 (+769283us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:31.019527 (+ 72us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:31.019585 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.019672 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.019831 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":275195,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":1600915,"threads_started":2}
W20260927 19:38:31.048041 2652 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.028s sys 0.030s
W20260927 19:38:31.048835 2652 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.783s user 0.028s sys 0.031s
I20260927 19:38:30.922369 3907 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:30.940234 3888 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:31.022130 3818 consensus_meta.cc:287] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 1.065s user 0.011s sys 0.009s
W20260927 19:38:31.022130 3574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.017s sys 0.049s
I20260927 19:38:31.019500 3542 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:31.041169 3794 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.137s user 0.030s sys 0.075s
I20260927 19:38:31.060982 2068 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:30.956040 3724 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:30.955271 3905 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:31.069835 2110 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.007s sys 0.019s
W20260927 19:38:31.071889 2110 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.008s sys 0.019s
I20260927 19:38:31.050251 3735 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:31.106491 2108 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.024s sys 0.010s
W20260927 19:38:31.066425 3574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.072s user 0.018s sys 0.049s
I20260927 19:38:31.073571 2142 raft_consensus.cc:595] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:38:31.124143 2108 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s user 0.024s sys 0.010s
W20260927 19:38:31.114540 1961 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.557s (SENT) [suppressed 7 similar messages]
I20260927 19:38:30.871724 2136 rpcz_store.cc:276] 0927 19:38:29.426948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.428639 (+ 1691us) service_pool.cc:227] Handling call
0927 19:38:29.439393 (+ 10754us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:29.465917 (+ 26524us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.468842 (+ 2925us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:29.527823 (+ 58981us) log.cc:842] Serialized 26 byte log entry
0927 19:38:29.867580 (+339757us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:29.870221 (+ 2641us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:29.890512 (+ 20291us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:30.871061 (+980549us) raft_consensus.cc:1671] finished
0927 19:38:30.871081 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:30.871259 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8758,"raft.queue_time_us":11003,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7827,"spinlock_wait_cycles":1931136,"thread_start_us":292110,"threads_started":1,"wal-append.queue_time_us":641611}
W20260927 19:38:31.075803 2679 consensus_meta.cc:287] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.852s user 0.010s sys 0.001s
I20260927 19:38:31.108373 1987 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.117234 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1373) took 1329 ms. Trace:
I20260927 19:38:31.018561 3930 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.068334 3818 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
W20260927 19:38:31.117516 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.005s sys 0.031s
I20260927 19:38:31.116953 3794 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.129068 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.131855 1996 rpcz_store.cc:276] 0927 19:38:29.787767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.796724 (+ 8957us) service_pool.cc:227] Handling call
0927 19:38:29.797174 (+ 450us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:29.845321 (+ 48147us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.845368 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:29.850326 (+ 4958us) log.cc:842] Serialized 26 byte log entry
0927 19:38:30.892669 (+1042343us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:30.892710 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:30.892787 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.117005 (+224218us) raft_consensus.cc:1671] finished
0927 19:38:31.117033 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.117175 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47719,"raft.queue_time_us":51793,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":4770,"thread_start_us":1020345,"threads_started":1,"wal-append.queue_time_us":1114180}
I20260927 19:38:31.133603 3824 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:31.137344 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.006s sys 0.031s
I20260927 19:38:31.155962 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:31.070843 3542 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:31.161587 2651 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.018s sys 0.024s
W20260927 19:38:31.175797 2651 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.823s user 0.018s sys 0.025s
W20260927 19:38:31.174352 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45224.4us
I20260927 19:38:31.177542 2037 leader_election.cc:398] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4.
I20260927 19:38:31.176363 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:31.179885 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.131371 2679 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:31.204262 2111 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.203650 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.172518 3906 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:31.211584 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:31.208604 3956 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260927 19:38:31.134286 3871 consensus_queue.cc:1046] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260927 19:38:31.209882 3966 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:31.202361 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.178745 3824 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.155620 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:60286 (request call id 1371) took 1378 ms. Trace:
I20260927 19:38:31.208720 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.177843 2068 raft_consensus.cc:2354] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5.
I20260927 19:38:31.207244 3574 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:31.215178 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.216807 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1818) took 1394 ms. Trace:
I20260927 19:38:31.130064 3735 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.172515 3794 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:31.182631 3944 log.cc:924] Time spent T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.473s user 0.010s sys 0.001s
W20260927 19:38:31.183454 2138 consensus_meta.cc:287] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.226s user 0.014s sys 0.002s
I20260927 19:38:31.180212 3962 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.180042 3869 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:31.219246 2066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1812) took 1974 ms (client timeout 2565 ms). Trace:
W20260927 19:38:31.300479 2066 rpcz_store.cc:269] 0927 19:38:29.239332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.240301 (+ 969us) service_pool.cc:227] Handling call
0927 19:38:31.213150 (+1972849us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b5400214654. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:31.213316 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3726415104}
W20260927 19:38:31.242731 3948 log.cc:924] Time spent T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.108s user 0.001s sys 0.001s
I20260927 19:38:31.180696 3872 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.219105 3776 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:31.244395 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1367) took 1583 ms. Trace:
W20260927 19:38:31.242182 3950 log.cc:924] Time spent T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
W20260927 19:38:31.245914 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 430945us
W20260927 19:38:31.312839 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 288515us
W20260927 19:38:31.267867 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 59226.9us
I20260927 19:38:31.320577 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.181171 3866 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.138558 3830 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:31.269541 3957 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260927 19:38:31.217756 2679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34930 (request call id 380) took 1205 ms. Trace:
I20260927 19:38:31.342834 2144 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:38:31.220952 3879 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.221446 3871 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.245608 3888 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.308115 1994 rpcz_store.cc:276] 0927 19:38:29.660897 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.661106 (+ 209us) service_pool.cc:227] Handling call
0927 19:38:29.672589 (+ 11483us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:29.716155 (+ 43566us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.716235 (+ 80us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:29.716657 (+ 422us) log.cc:842] Serialized 26 byte log entry
0927 19:38:29.826449 (+109792us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b180021d5a0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e9411 00007f79139365cf
0927 19:38:31.027101 (+1200652us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:31.027156 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.027245 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.244090 (+216845us) raft_consensus.cc:1671] finished
0927 19:38:31.244111 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.244338 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43104,"raft.queue_time_us":44391,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":12076,"spinlock_wait_cycles":7578368,"thread_start_us":1200447,"threads_started":1,"wal-append.queue_time_us":1462812}
I20260927 19:38:31.298422 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.334842 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:31.339339 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.344723 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:31.387081 2140 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:31.387784 2144 raft_consensus.cc:3032] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:31.388094 2144 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:31.346565 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.294661 3852 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.297706 3745 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.307338 3905 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.363847 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:31.368459 3945 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.588s user 0.004s sys 0.003s
I20260927 19:38:31.370303 2145 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:31.369935 3924 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:31.371942 1990 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1341) took 2249 ms (client timeout 2144 ms). Trace:
W20260927 19:38:31.372807 3946 log.cc:924] Time spent T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.306s user 0.000s sys 0.000s
I20260927 19:38:31.375962 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:56794 (request call id 14) took 1659 ms. Trace:
W20260927 19:38:31.376817 2065 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1816) took 1555 ms (client timeout 1557 ms). Trace:
W20260927 19:38:31.383488 3954 log.cc:924] Time spent T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.204s user 0.003s sys 0.004s
I20260927 19:38:31.267087 3772 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260927 19:38:31.268510 3906 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260927 19:38:31.299368 2138 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:31.240926 3885 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:31.267215 1987 rpcz_store.cc:276] 0927 19:38:29.784940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.786443 (+ 1503us) service_pool.cc:227] Handling call
0927 19:38:31.135048 (+1348605us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260927 19:38:31.352854 3942 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.534s user 0.000s sys 0.001s
I20260927 19:38:31.402931 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:31.453660 2139 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:31.266971 3824 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.270484 3966 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:31.388881 2144 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.310072 3907 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.340147 3724 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.408596 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.309526 3962 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.414140 2142 rpcz_store.cc:276] 0927 19:38:29.716931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.717176 (+ 245us) service_pool.cc:227] Handling call
0927 19:38:31.375726 (+1658550us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002bb6d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f791395cbc3 00007f79188a681a 00007f7911f99d04 00007f7911f99c92 00007f7911f99bc1 00007f7911f99b3c 00007f7911f98df2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:31.375846 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3093745920}
I20260927 19:38:31.417825 2143 raft_consensus.cc:1226] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:31.394088 3965 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.396307 3869 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.428421 3776 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:31.413662 3955 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.301s user 0.001s sys 0.000s
I20260927 19:38:31.474062 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260927 19:38:31.474259 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 262509us
I20260927 19:38:31.440003 3542 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.475766 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:31.397007 3970 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:31.403786 3866 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.407265 3334 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.404824 3879 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.456512 2144 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:31.495635 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.496347 2143 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:31.346035 2679 rpcz_store.cc:276] 0927 19:38:30.006148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.020668 (+ 14520us) service_pool.cc:227] Handling call
0927 19:38:31.207536 (+1186868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:31.339663 3964 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.134s
I20260927 19:38:31.410995 3888 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.460321 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.504021 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:31.504267 2145 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:31.411514 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:31.462110 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 347581us
I20260927 19:38:31.344818 3972 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.142s
I20260927 19:38:31.418915 3772 consensus_queue.cc:1225] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer 3d1bc2cd9efe43399ddba396f5ddff85 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260927 19:38:31.443819 3952 log.cc:924] Time spent T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.331s user 0.000s sys 0.001s
I20260927 19:38:31.427475 3778 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4.
I20260927 19:38:31.475455 3968 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.398319 3871 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.445214 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1020) took 1705 ms. Trace:
I20260927 19:38:31.478330 3885 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.478888 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.523017 1991 rpcz_store.cc:276] 0927 19:38:29.739960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.740084 (+ 124us) service_pool.cc:227] Handling call
0927 19:38:29.744828 (+ 4744us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:29.756454 (+ 11626us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.756486 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:29.756915 (+ 429us) log.cc:842] Serialized 26 byte log entry
0927 19:38:29.910196 (+153281us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b4800271a58. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79188097f1 00007f79188008e7 00007f791880077a 00007f79188f495f 00007f79188f4627 00007f7918908b2c 00007f7918908ad8 00007f7918908a58 00007f7918908a13 00007f7918907cbc 00007f7917889be6
0927 19:38:30.148490 (+238294us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:30.148551 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:30.148649 (+ 98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.405950 (+1257301us) raft_consensus.cc:1671] finished
0927 19:38:31.405978 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.445130 (+ 39152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11279,"raft.queue_time_us":39098,"raft.run_cpu_time_us":2540,"raft.run_wall_time_us":123404,"spinlock_wait_cycles":5564800,"thread_start_us":354572,"threads_started":1,"wal-append.queue_time_us":738743}
I20260927 19:38:31.526768 2142 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:31.528116 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.529829 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:31.457094 3924 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:31.534873 3907 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:31.345535 3872 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.537400 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.538961 3907 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:31.540429 3872 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.484248 3833 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:31.541565 3907 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:31.484764 2139 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:31.485659 3432 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:31.488735 2066 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:38:31.547636 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:31.547168 3907 consensus_queue.cc:1046] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:31.490330 2686 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
W20260927 19:38:31.473575 3779 consensus_meta.cc:287] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 1.441s user 0.015s sys 0.000s
W20260927 19:38:31.456938 2138 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46930 (request call id 250) took 1607 ms (client timeout 1606 ms). Trace:
I20260927 19:38:31.497982 3966 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.504835 2068 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:31.505405 2141 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
W20260927 19:38:31.419474 2065 rpcz_store.cc:269] 0927 19:38:29.821851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.833726 (+ 11875us) service_pool.cc:227] Handling call
0927 19:38:31.376578 (+1542852us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b5400324954. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:31.376750 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21431552}
W20260927 19:38:31.495383 3963 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
I20260927 19:38:31.516338 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.417341 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:31.517158 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 124651us
W20260927 19:38:31.413926 1990 rpcz_store.cc:269] 0927 19:38:29.122950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.132707 (+ 9757us) service_pool.cc:227] Handling call
0927 19:38:29.750579 (+617872us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:31.371678 (+1621099us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54000e92d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:31.371844 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313177344}
I20260927 19:38:31.521821 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.427125 3797 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.549053 3905 raft_consensus.cc:3032] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:31.560441 3905 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:31.560757 3797 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.553164 2144 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:31.549106 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:31.562336 3905 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.564064 2145 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:31.564491 3905 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:31.565519 2066 raft_consensus.cc:3032] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:31.565874 2066 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:31.566927 2066 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:31.564859 2138 rpcz_store.cc:269] 0927 19:38:29.845280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.848998 (+ 3718us) service_pool.cc:227] Handling call
0927 19:38:31.441302 (+1592304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:31.567508 2688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 355) took 1434 ms. Trace:
I20260927 19:38:31.567840 2066 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:31.568096 2688 rpcz_store.cc:276] 0927 19:38:30.132734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.132870 (+ 136us) service_pool.cc:227] Handling call
0927 19:38:30.170802 (+ 37932us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:30.185629 (+ 14827us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:30.185672 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:30.208503 (+ 22831us) log.cc:842] Serialized 52 byte log entry
0927 19:38:31.015107 (+806604us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:31.015170 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.015272 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.567251 (+551979us) raft_consensus.cc:1671] finished
0927 19:38:31.567275 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.567421 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":837,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":9609,"thread_start_us":693418,"threads_started":1,"wal-append.queue_time_us":910296}
I20260927 19:38:31.560305 3779 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4.
I20260927 19:38:31.541097 3872 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.572764 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.583627 3779 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:31.584165 3779 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.585021 3779 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:31.562937 3797 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.589401 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:31.601058 3745 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:31.601622 3745 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.602046 3745 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:31.516739 3960 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.303s user 0.001s sys 0.000s
W20260927 19:38:31.629819 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 68283.1us
I20260927 19:38:31.624245 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.634537 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.635725 2653 leader_election.cc:398] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3.
I20260927 19:38:31.636889 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:31.644115 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 111747us
W20260927 19:38:31.648226 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 172250us
I20260927 19:38:31.652570 2138 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 19:38:31.474777 3869 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.515784 3906 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.521591 3970 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:31.619573 3958 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.435s user 0.000s sys 0.001s
I20260927 19:38:31.648433 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:31.667966 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:31.622726 3983 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.649195 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:31.522491 3908 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.479772 3776 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:31.662209 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 137530us
I20260927 19:38:31.665300 2064 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:31.496156 3866 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.474855 3965 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.657228 3728 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.574803 3872 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.654659 3852 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:31.633206 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.515676 3888 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.654507 3830 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.657121 3964 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.701287 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.657446 3794 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:31.657269 3735 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.710609 3735 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.711077 3735 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.712088 3975 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:31.713076 3735 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.705907 3830 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.715157 3866 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.715667 3866 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.716257 3866 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.664892 3824 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.688387 3983 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.669054 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:31.522820 3879 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.659241 3724 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.720755 3983 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.669427 3818 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.724069 2142 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:31.727375 2144 raft_consensus.cc:1226] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:31.665455 3985 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:31.668181 3974 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.629536 3885 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.701960 2685 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:31.619371 3968 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.701637 3964 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.748109 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1825) took 1923 ms. Trace:
I20260927 19:38:31.678805 3542 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.709286 3888 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.715716 3830 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.501686 3334 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.720865 3724 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.724215 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 189) took 1111 ms. Trace:
I20260927 19:38:31.784472 2108 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.689646 3908 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.721489 3879 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.726212 3818 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.741335 3871 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.699386 3772 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.741716 3974 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.742452 3869 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.703263 3972 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.271031 2062 rpcz_store.cc:276] 0927 19:38:29.822416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.826842 (+ 4426us) service_pool.cc:227] Handling call
0927 19:38:29.968313 (+141471us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b5800043598. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:29.968530 (+ 217us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:29.968959 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.968991 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:29.969634 (+ 643us) log.cc:842] Serialized 52 byte log entry
0927 19:38:30.923209 (+953575us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:30.928454 (+ 5245us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:30.938973 (+ 10519us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.216466 (+277493us) raft_consensus.cc:1671] finished
0927 19:38:31.216485 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.216727 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1192,"raft.queue_time_us":1237,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":5884,"spinlock_wait_cycles":13044480,"thread_start_us":931810,"threads_started":1,"wal-append.queue_time_us":957879}
I20260927 19:38:31.748685 2070 rpcz_store.cc:276] 0927 19:38:29.824278 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.844083 (+ 19805us) service_pool.cc:227] Handling call
0927 19:38:29.844337 (+ 254us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:29.844659 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.844706 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:29.852635 (+ 7929us) log.cc:842] Serialized 26 byte log entry
0927 19:38:30.900492 (+1047857us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:30.929338 (+ 28846us) log.cc:842] Serialized 14 byte log entry
0927 19:38:30.929633 (+ 295us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:30.929718 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.747906 (+818188us) raft_consensus.cc:1671] finished
0927 19:38:31.747928 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.748052 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28622,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":10186,"raft.queue_time_us":9173,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":21656,"thread_start_us":1003282,"threads_started":1,"wal-append.queue_time_us":1185534}
I20260927 19:38:31.761924 3852 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.767293 2140 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:31.756528 3985 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.784827 2067 rpcz_store.cc:276] 0927 19:38:30.698446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.808379 (+109933us) service_pool.cc:227] Handling call
0927 19:38:30.808761 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:30.862479 (+ 53718us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:30.862519 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:30.862570 (+ 51us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:30.862609 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.723998 (+861389us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b5400323a54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:31.724047 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.724166 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123019520}
I20260927 19:38:31.666142 3906 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.744040 3872 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.744009 3866 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.737633 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.697281 3728 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.789682 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.790850 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.744035 3885 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.676051 3970 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.826582 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.791455 3981 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:31.866195 3983 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.869227 3981 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.862311 3905 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.869771 3981 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:31.870195 3905 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.871781 3334 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.872260 3905 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.834579 3965 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.844843 3985 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.847448 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.849004 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.849740 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:31.851878 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.854676 3866 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.784777 3992 consensus_queue.cc:1046] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260927 19:38:31.782814 3542 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:31.858417 3941 log.cc:924] Time spent T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.993s user 0.001s sys 0.007s
W20260927 19:38:31.861094 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 71515.3us
W20260927 19:38:31.863469 3947 log.cc:924] Time spent T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 1.054s user 0.010s sys 0.002s
I20260927 19:38:31.863704 3776 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.864123 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:31.869714 3724 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.868211 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:31.838618 3972 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.874372 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:31.884162 2684 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:31.869510 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.869508 3983 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.876252 3818 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.876398 1987 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:31.876477 3974 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:31.827961 3772 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.886650 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.869249 3978 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:31.864264 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:31.873426 3962 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:31.903451 3368 consensus_meta.cc:287] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.994s user 0.003s sys 0.010s
I20260927 19:38:31.868429 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.839056 3852 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.894460 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:31.894742 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.837424 3879 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:31.901698 1994 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260927 19:38:31.901676 1962 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.907773 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.891176 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.915987 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 188) took 1380 ms. Trace:
W20260927 19:38:31.892010 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 241150us
I20260927 19:38:31.886687 3924 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260927 19:38:31.919447 2144 raft_consensus.cc:2362] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4. Re-sending same reply.
I20260927 19:38:31.921594 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.926575 2063 rpcz_store.cc:276] 0927 19:38:30.536054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.606794 (+ 70740us) service_pool.cc:227] Handling call
0927 19:38:30.607228 (+ 434us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:31.062380 (+455152us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:31.062429 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:31.062489 (+ 60us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:31.062531 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.915785 (+853254us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54000fd554. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:31.915832 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.915936 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1046107904}
I20260927 19:38:31.906368 3905 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.886009 3866 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:31.929504 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.946516 2654 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.948374 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:31.929062 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.960505 2140 raft_consensus.cc:2396] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:31.964831 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:31.968044 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:31.969485 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52970.9us
I20260927 19:38:31.970304 1961 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:31.971626 1962 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.906406 3779 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.923477 3980 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:31.940737 3829 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.067s user 0.008s sys 0.002s
I20260927 19:38:31.922523 3869 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:31.925930 3871 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:31.964434 3906 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.974988 4013 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260927 19:38:31.977841 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1372) took 2185 ms. Trace:
I20260927 19:38:31.929555 3875 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:31.947929 3968 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:31.925998 3830 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:31.963410 3885 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260927 19:38:31.926656 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:32.003233 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 366372us
I20260927 19:38:31.975848 1962 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:31.974542 2110 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:32.019435 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 45025.9us
I20260927 19:38:32.020154 2110 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:31.976694 3962 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:31.976457 3994 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:31.928246 3879 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:32.022909 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 47194.6us
I20260927 19:38:32.005120 1993 rpcz_store.cc:276] 0927 19:38:29.787473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.794308 (+ 6835us) service_pool.cc:227] Handling call
0927 19:38:29.794611 (+ 303us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:29.819858 (+ 25247us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:29.819889 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:29.820319 (+ 430us) log.cc:842] Serialized 26 byte log entry
0927 19:38:30.244436 (+424117us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:30.244487 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:30.244583 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:31.859577 (+1614994us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:31.958747 (+ 99170us) raft_consensus.cc:1671] finished
0927 19:38:31.958777 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:31.971093 (+ 12316us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23468,"raft.queue_time_us":23676,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"spinlock_wait_cycles":286334848,"thread_start_us":423743,"threads_started":1,"wal-append.queue_time_us":712548}
I20260927 19:38:32.027450 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.028445 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.028717 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:31.956830 3792 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260927 19:38:31.971175 3966 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.055768 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.056977 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.016448 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:31.981460 3964 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:31.980997 3888 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
W20260927 19:38:31.983973 3799 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260927 19:38:31.926785 3368 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:31.986130 3988 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
W20260927 19:38:31.983247 3829 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.108s user 0.009s sys 0.003s
I20260927 19:38:31.966449 3905 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.029022 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.007369 3875 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:31.980865 3574 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260927 19:38:32.015401 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.018126 3728 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.021518 3906 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:31.997987 3869 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.025408 3794 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.060835 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1386) took 1829 ms. Trace:
I20260927 19:38:32.029912 3968 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:32.087009 3968 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.064808 2653 leader_election.cc:398] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5.
I20260927 19:38:32.089488 2653 leader_election.cc:401] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260927 19:38:32.066138 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.061024 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:32.004331 3871 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:32.068059 4003 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260927 19:38:32.030329 3542 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:31.986405 3980 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.070098 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.051761 4013 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:32.049914 3962 raft_consensus.cc:686] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:32.077777 2145 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:32.098773 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.098888 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.053673 3970 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.082477 1991 raft_consensus.cc:1226] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:32.099586 2139 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:32.084924 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.084625 1989 rpcz_store.cc:276] 0927 19:38:30.231001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.912000 (+680999us) service_pool.cc:227] Handling call
0927 19:38:30.912286 (+ 286us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:30.919361 (+ 7075us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 19:38:31.301142 (+381781us) log.cc:842] Serialized 14 byte log entry
0927 19:38:31.926581 (+625439us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:31.926980 (+ 399us) log.cc:842] Serialized 26 byte log entry
0927 19:38:31.927114 (+ 134us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:31.927148 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:31.927205 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:32.059678 (+132473us) raft_consensus.cc:1671] finished
0927 19:38:32.059701 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:32.060778 (+ 1077us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23216,"prepare.queue_time_us":522425,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":226694,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":7404,"thread_start_us":968339,"threads_started":3,"wal-append.queue_time_us":416527}
I20260927 19:38:32.074918 3888 raft_consensus.cc:686] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:32.111778 2681 consensus_meta.cc:287] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.685s user 0.007s sys 0.003s
I20260927 19:38:32.111315 3888 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.112286 2681 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:32.091956 3972 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.060956 3772 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:32.064652 3735 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.114084 3735 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.115545 1962 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:32.116128 3735 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.116873 3735 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:32.083909 3985 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.073490 3797 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:32.018795 3830 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.109185 4012 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.087s
I20260927 19:38:32.119359 3830 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.102396 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.120591 1989 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:32.120970 3871 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.082132 3875 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.080668 3908 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.093979 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.097777 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.105968 3989 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5.
I20260927 19:38:32.106452 2145 raft_consensus.cc:1226] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260927 19:38:32.108765 3997 log.cc:924] Time spent T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.175s user 0.001s sys 0.000s
I20260927 19:38:32.110293 4017 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:32.139494 4017 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:32.140259 4001 consensus_queue.cc:1046] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:32.079347 3988 raft_consensus.cc:686] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:32.141950 4013 raft_consensus.cc:686] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:32.142261 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.142486 3988 consensus_queue.cc:237] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.142833 4013 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:32.113008 2681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34940 (request call id 354) took 2151 ms (client timeout 1878 ms). Trace:
I20260927 19:38:32.125361 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.127455 1959 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.149773 1959 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:32.131742 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 110541us
I20260927 19:38:32.134022 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.134749 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:32.152432 2063 raft_consensus.cc:2362] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4. Re-sending same reply.
I20260927 19:38:32.134865 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.140967 4017 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.117239 3735 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:32.123287 3852 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.131090 3970 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.142342 3905 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.160409 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 190) took 1450 ms. Trace:
I20260927 19:38:32.162437 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.162156 2069 rpcz_store.cc:276] 0927 19:38:30.710319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.784794 (+ 74475us) service_pool.cc:227] Handling call
0927 19:38:30.785131 (+ 337us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:31.051451 (+266320us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:31.051530 (+ 79us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:31.051585 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:31.051626 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:32.160210 (+1108584us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54002756d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:32.160251 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:32.160358 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":611885440}
I20260927 19:38:32.159885 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.131805 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.144861 3968 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.109131 2684 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:32.168390 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.165226 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.176918 4009 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.177122 4007 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:32.155803 2136 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:32.161018 3735 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:32.156118 3779 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.161307 4004 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.178706 2110 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:32.148213 2681 rpcz_store.cc:269] 0927 19:38:29.961790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.965466 (+ 3676us) service_pool.cc:227] Handling call
0927 19:38:31.363627 (+1398161us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:32.112905 (+749278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10535808}
I20260927 19:38:32.173698 3368 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.183039 3879 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.159894 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.160554 3772 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260927 19:38:32.177553 3968 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:32.173080 3965 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.018s
I20260927 19:38:32.184408 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.200049 3968 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:32.200433 3968 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:32.200783 3968 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:32.174887 3745 consensus_meta.cc:287] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.570s user 0.005s sys 0.013s
I20260927 19:38:32.165072 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.168133 3852 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.175176 3574 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
I20260927 19:38:32.185388 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:32.190919 3908 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:32.207238 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.183777 3905 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20260927 19:38:32.179443 3794 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260927 19:38:32.168284 3970 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.204034 3869 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.195928 3866 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.197131 2062 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:32.183921 3962 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.215940 4002 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:32.218623 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:32.222193 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.222362 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.222858 2063 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:32.222950 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.225498 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.228165 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.229621 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.229808 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:32.230121 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.190721 4009 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.233999 3906 raft_consensus.cc:686] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:32.232456 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.233279 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.234740 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.211323 2138 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:32.230263 2063 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:32.231182 2067 raft_consensus.cc:2396] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:32.208870 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.259225 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.242297 2653 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:32.261629 2066 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:32.261032 2070 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:32.204512 3745 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.264065 1996 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:32.270712 4009 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:32.235395 3980 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.272030 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.272439 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:32.266891 3890 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.277927 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.279557 3852 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.281822 3852 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.282500 3852 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.272969 3993 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:32.279492 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:32.240236 3978 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.291280 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.292325 3745 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.295192 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:32.298163 3987 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.540s user 0.004s sys 0.000s
I20260927 19:38:32.302402 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.302628 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.304183 4024 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:32.304960 3813 raft_consensus.cc:3032] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:32.280818 3890 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.288808 3735 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.298466 3852 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.322043 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.322649 2067 raft_consensus.cc:1226] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:32.319733 3813 raft_consensus.cc:729] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:32.300663 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1377) took 2389 ms. Trace:
I20260927 19:38:32.322753 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.321920 3890 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.330376 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.330668 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.330408 3906 consensus_queue.cc:237] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.325558 3985 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.303819 3994 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:32.337679 4024 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.303369 3981 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.346580 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:32.348241 2063 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:32.327739 3838 consensus_queue.cc:1046] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260927 19:38:32.348483 2110 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.355711 2110 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:32.356624 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.342919 3970 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.347618 4008 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.362542 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.364812 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.326431 1988 rpcz_store.cc:276] 0927 19:38:29.911536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:29.922899 (+ 11363us) service_pool.cc:227] Handling call
0927 19:38:29.923148 (+ 249us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:31.021230 (+1098082us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:32.276726 (+1255496us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b540031c254. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:32.298964 (+ 22238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2188709248}
I20260927 19:38:32.368781 2110 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.372900 2110 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:32.362265 3965 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.374581 3833 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:32.361086 3886 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:32.377699 4022 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.388795 3833 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.379792 3994 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.382035 3735 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.365036 3792 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.389780 2140 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:32.366513 3995 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.390712 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:32.399974 2145 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:32.371671 3574 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:32.392022 3985 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:32.373737 3980 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.388433 3965 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:32.386284 3875 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.392750 3833 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:32.392226 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:32.376160 2110 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:32.402195 3980 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:32.416662 2110 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:32.403209 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.407378 3965 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.462945 4024 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:32.474606 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.476291 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.402433 3735 consensus_queue.cc:1046] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260927 19:38:32.479697 3735 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:32.373497 4008 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.480192 3735 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:32.480779 1996 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:32.392212 4022 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.414049 3574 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:32.486312 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.490696 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.466477 3964 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.415627 3980 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.488919 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.406857 3985 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.534791 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.468015 3970 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.514923 2145 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:32.517838 3968 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.486882 3574 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.482299 3981 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.519474 3866 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:32.465601 3852 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.486734 4022 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.541848 1962 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.535714 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.521265 3964 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.534314 3888 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:32.568744 2063 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260927 19:38:32.539695 3728 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.542677 3574 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:32.576738 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.546031 3993 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.579466 1994 raft_consensus.cc:2396] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:32.570787 3964 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.481289 3735 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.568567 3887 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:32.590837 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.591742 1994 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:32.594187 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.595721 2037 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.542838 3970 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.600925 1959 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:32.603601 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.604445 1994 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:32.605177 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.605545 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:32.606608 1994 raft_consensus.cc:2354] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:32.482532 4024 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.608636 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.614349 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.621234 1994 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:32.619537 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:32.621560 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:32.574491 3908 raft_consensus.cc:686] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:32.602568 3866 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:32.623723 2037 leader_election.cc:398] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3.
I20260927 19:38:32.626477 2037 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.628654 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:32.624850 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:32.631316 1994 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.574486 3888 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.608295 4008 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.608130 3887 raft_consensus.cc:686] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:32.624378 3988 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:32.587626 3728 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.623493 3875 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.588610 3993 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.625146 3735 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:32.625114 3866 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.633337 3869 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:32.631001 4007 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:32.677600 1988 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:32.729681 4007 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.718520 3745 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:32.721481 3866 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:32.766276 3993 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:32.745450 4011 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.765s user 0.004s sys 0.001s
I20260927 19:38:32.768472 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 261) took 1282 ms. Trace:
I20260927 19:38:32.748315 4007 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:32.755275 4025 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.257s user 0.002s sys 0.000s
I20260927 19:38:32.744117 3968 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.813706 1994 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:32.814998 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.743291 3985 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.815904 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.816576 1994 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
W20260927 19:38:32.757027 4038 log.cc:924] Time spent T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.172s user 0.000s sys 0.002s
I20260927 19:38:32.817546 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.818842 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:32.820258 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.821095 1994 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:32.821877 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.822454 1994 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:32.823227 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.824265 1994 raft_consensus.cc:2396] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:32.742429 3908 consensus_queue.cc:237] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.837347 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.824636 3879 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:32.839871 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:32.826892 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:32.841271 1994 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:32.829736 4026 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:32.829465 2684 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:32.847252 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:32.836705 3970 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.849730 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:32.851320 1994 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:32.840150 3830 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:32.856593 3830 raft_consensus.cc:686] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:32.857815 3830 consensus_queue.cc:237] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.846983 3879 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.848176 2142 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:32.800984 3887 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.861115 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
W20260927 19:38:32.861405 2064 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1356) took 2671 ms (client timeout 1816 ms). Trace:
I20260927 19:38:32.805933 2137 rpcz_store.cc:276] 0927 19:38:31.497563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.560745 (+ 63182us) service_pool.cc:227] Handling call
0927 19:38:31.561061 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:31.714916 (+153855us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:31.714949 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:31.714994 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:31.715025 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:32.768109 (+1053084us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54000762d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:32.768205 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:32.768407 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":333790592}
W20260927 19:38:32.872090 2064 rpcz_store.cc:269] 0927 19:38:30.181295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:30.182021 (+ 726us) service_pool.cc:227] Handling call
0927 19:38:32.852180 (+2670159us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002a4254. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:32.852348 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3126605056}
I20260927 19:38:32.883001 2069 pending_rounds.cc:76] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 1
I20260927 19:38:32.883376 2069 pending_rounds.cc:97] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260927 19:38:32.883667 2069 raft_consensus.cc:2866] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:32.885432 3852 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:32.881176 4042 log.cc:924] Time spent T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260927 19:38:32.854586 4026 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:32.885913 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:32.922986 3972 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:32.923624 3972 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.867034 3968 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:32.925972 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
W20260927 19:38:32.891708 4041 log.cc:924] Time spent T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260927 19:38:32.868451 3888 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:32.881829 3985 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260927 19:38:32.900326 1994 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:32.948557 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.049083 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.086020 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:33.090636 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54637.6us
W20260927 19:38:33.090940 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 97934us
I20260927 19:38:32.891667 4026 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:33.019647 2138 consensus_meta.cc:287] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.750s user 0.009s sys 0.003s
I20260927 19:38:32.888212 4024 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:33.043110 3890 consensus_meta.cc:287] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.703s user 0.008s sys 0.004s
W20260927 19:38:33.031629 3833 consensus_meta.cc:287] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.616s user 0.011s sys 0.000s
I20260927 19:38:32.942785 2062 raft_consensus.cc:1226] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:32.969883 3965 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:32.932240 3968 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:32.937574 3779 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:32.893692 4044 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:33.110553 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.892028 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.886117 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:32.931751 3728 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.969482 3972 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:32.934273 3970 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:33.119015 2138 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:32.933961 3983 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.012173 3574 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.041473 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 255) took 1084 ms. Trace:
I20260927 19:38:33.117101 1989 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:33.120468 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:32.957042 3977 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.160s
I20260927 19:38:32.976840 3980 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.166666 2139 raft_consensus.cc:2379] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:33.033175 3879 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.172966 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.174559 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.210743 2141 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:33.174892 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46930 (request call id 267) took 1295 ms. Trace:
I20260927 19:38:33.175603 3866 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.176966 3965 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:33.177645 3964 consensus_meta.cc:287] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.574s user 0.005s sys 0.005s
I20260927 19:38:33.166373 3887 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:33.129055 3969 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:33.130707 3779 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:33.233707 3779 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:33.191565 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 143672us
I20260927 19:38:33.190829 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 1981) took 1014 ms. Trace:
I20260927 19:38:33.135880 4044 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:33.126281 3833 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.125241 3890 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.215642 3980 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:33.215364 2136 rpcz_store.cc:276] 0927 19:38:31.879260 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.880360 (+ 1100us) service_pool.cc:227] Handling call
0927 19:38:33.174720 (+1294360us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400051254. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:33.174823 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514938240}
I20260927 19:38:33.131295 3968 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:33.239472 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.259334 1994 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:33.238399 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:33.260438 2064 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.261379 2145 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:33.240958 3887 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.221560 4007 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.246940 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 1035) took 1216 ms. Trace:
I20260927 19:38:33.264226 1988 rpcz_store.cc:276] 0927 19:38:32.030127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.534554 (+504427us) service_pool.cc:227] Handling call
0927 19:38:33.246729 (+712175us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540031bd54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:33.246871 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300804352}
I20260927 19:38:33.242287 2037 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:33.251084 4044 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.254922 3908 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:33.251116 3972 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:33.267796 3977 consensus_queue.cc:1046] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260927 19:38:33.275228 3977 raft_consensus.cc:3032] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:33.275502 3887 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.274768 3972 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.273850 2687 raft_consensus.cc:1226] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:33.275570 3977 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:33.269482 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:33.281996 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:33.252348 3818 consensus_queue.cc:1046] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260927 19:38:33.283888 3818 raft_consensus.cc:3032] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:33.284269 3818 raft_consensus.cc:729] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:33.287624 4007 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.289206 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:33.290019 3970 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:33.290393 3977 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.291838 3977 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:33.292467 3866 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.240737 2138 rpcz_store.cc:276] 0927 19:38:32.176075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.177373 (+ 1298us) service_pool.cc:227] Handling call
0927 19:38:33.189153 (+1011780us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:33.293861 3574 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.278798 3985 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:33.306844 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:33.296617 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 1.28379e+06us
I20260927 19:38:33.280571 3908 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.298418 3907 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:33.285014 3930 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:33.280967 3833 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.299154 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:33.305449 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.305924 2684 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:33.191900 1990 rpcz_store.cc:276] 0927 19:38:31.957069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.963563 (+ 6494us) service_pool.cc:227] Handling call
0927 19:38:31.999664 (+ 36101us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:32.072248 (+ 72584us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:32.074855 (+ 2607us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:32.078664 (+ 3809us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:32.084122 (+ 5458us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.041098 (+956976us) spinlock_profiling.cc:243] Waited 60.1 ms on lock 0x7b54000e88d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:33.041172 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:33.041390 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138168576}
I20260927 19:38:33.307735 1962 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:33.308306 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:33.234726 3964 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.310386 3970 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:33.310570 4010 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:33.313854 4017 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.314575 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:33.309310 3985 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:33.285087 3818 consensus_queue.cc:260] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.327396 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:33.341482 3907 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:33.337584 2137 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:33.338449 2681 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260927 19:38:33.347653 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.348330 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.349915 4008 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:33.246997 3969 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.330508 2064 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:33.327463 3866 consensus_queue.cc:1046] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260927 19:38:33.334457 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1368) took 1844 ms. Trace:
I20260927 19:38:33.329391 4007 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:33.350771 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:33.377275 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 49992.6us
I20260927 19:38:33.391584 2067 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.391727 2681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34930 (request call id 392) took 1197 ms. Trace:
I20260927 19:38:33.392139 2681 rpcz_store.cc:276] 0927 19:38:32.193989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.197322 (+ 3333us) service_pool.cc:227] Handling call
0927 19:38:33.298988 (+1101666us) spinlock_profiling.cc:243] Waited 983 us on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:33.391660 (+ 92672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2259840}
I20260927 19:38:33.355536 3735 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:33.393654 3735 raft_consensus.cc:686] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:33.394349 4028 log.cc:924] Time spent T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.912s user 0.003s sys 0.000s
I20260927 19:38:33.394642 3735 consensus_queue.cc:237] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.353346 4061 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:33.398221 4032 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.643s user 0.002s sys 0.000s
I20260927 19:38:33.358237 2142 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:33.359292 3985 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:33.381603 2686 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34930 (request call id 384) took 2172 ms (client timeout 1881 ms). Trace:
I20260927 19:38:33.357545 3998 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:33.389647 2071 rpcz_store.cc:276] 0927 19:38:31.489950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.490535 (+ 585us) service_pool.cc:227] Handling call
0927 19:38:32.852402 (+1361867us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:32.852888 (+ 486us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:32.852936 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:32.852990 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:32.853027 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:32.853113 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:33.334292 (+481179us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002a4250. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:33.334398 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":285,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":12650,"spinlock_wait_cycles":3131109760}
I20260927 19:38:33.401477 2110 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:33.402240 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:33.406723 2068 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:33.403137 2686 rpcz_store.cc:269] 0927 19:38:31.209298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.211328 (+ 2030us) service_pool.cc:227] Handling call
0927 19:38:33.381342 (+2170014us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b540061fdd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:33.381499 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":639355136}
I20260927 19:38:33.355757 3879 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260927 19:38:33.366291 3972 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:33.398083 4061 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.409940 3833 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:33.379843 3970 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:33.414724 3818 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:33.418802 3833 raft_consensus.cc:686] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:33.404037 4007 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.417995 3970 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:33.418941 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:33.424901 3908 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.429718 3833 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.416041 3779 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.417845 3998 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.428109 3970 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.447777 3813 consensus_queue.cc:260] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.449023 3813 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:33.437788 3964 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.428910 3907 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.453666 3978 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:33.454859 3978 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.489669 3912 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:33.454347 3969 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:33.559561 4061 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.547367 3998 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.568775 3968 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:33.571897 4070 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
W20260927 19:38:33.574115 4071 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.152s user 0.001s sys 0.000s
I20260927 19:38:33.579470 4007 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:33.585492 4072 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260927 19:38:33.452189 4044 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.589135 4044 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.590281 3985 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.592733 3985 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:33.351800 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44146us
I20260927 19:38:33.647425 1962 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:33.649737 1987 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:33.378285 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.655931 1990 raft_consensus.cc:1226] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:33.659049 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:33.376775 1994 raft_consensus.cc:2396] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:33.403939 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:33.709347 2065 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:33.719980 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.723721 3968 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.443562 3779 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.730170 3970 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.731359 3908 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:33.737983 3977 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:33.739157 3983 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:33.742869 2069 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:33.751545 3977 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:33.490304 3912 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.761941 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:33.756024 3977 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:33.771843 1988 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:33.777756 1993 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:33.779716 3983 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.789095 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:33.800647 2108 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:33.805827 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1395) took 1458 ms. Trace:
I20260927 19:38:33.805738 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:33.806295 3983 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:33.805449 2144 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 1976) took 1712 ms (client timeout 2093 ms). Trace:
I20260927 19:38:33.794991 3978 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.807523 2034 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:33.806402 2063 rpcz_store.cc:276] 0927 19:38:32.347028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.350616 (+ 3588us) service_pool.cc:227] Handling call
0927 19:38:32.350910 (+ 294us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:32.572622 (+221712us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.805086 (+1232464us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b540028dcd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:33.805616 (+ 530us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":500244224}
W20260927 19:38:33.807835 2144 rpcz_store.cc:269] 0927 19:38:32.082395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.090313 (+ 7918us) service_pool.cc:227] Handling call
0927 19:38:33.719552 (+1629239us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:33.794637 (+ 75085us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4652928}
I20260927 19:38:33.831188 2034 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:33.832175 4074 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:33.834182 4074 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.835307 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:33.836629 2063 raft_consensus.cc:2379] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:33.832975 3852 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:33.838294 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:33.838477 3852 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:33.838853 3890 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.627238 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:33.839881 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:33.852958 3872 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:33.854575 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:33.861936 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:33.866369 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 66000.8us
I20260927 19:38:33.851711 4074 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:33.847905 3852 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:33.878180 2681 raft_consensus.cc:3032] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:33.878830 2681 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:33.880096 2681 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.880606 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:33.881896 2681 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:33.878331 3852 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:33.882182 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.883157 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.883965 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.884063 3852 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:33.885953 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:33.886287 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:33.886869 2063 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:33.887003 2687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 579) took 1365 ms. Trace:
I20260927 19:38:33.887853 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:33.891816 3970 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:33.892454 1990 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:33.893462 1962 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:33.884150 3912 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:33.900842 1988 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
W20260927 19:38:33.892498 1993 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56618 (request call id 260) took 1878 ms (client timeout 1793 ms). Trace:
I20260927 19:38:33.826499 3908 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:33.877728 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:33.905855 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:33.906581 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:33.888665 2063 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:33.902343 1993 rpcz_store.cc:269] 0927 19:38:32.013859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.028485 (+ 14626us) service_pool.cc:227] Handling call
0927 19:38:33.892264 (+1863779us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b5400193f54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:33.892397 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4019984128}
I20260927 19:38:33.918198 1994 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:33.887454 2687 rpcz_store.cc:276] 0927 19:38:32.521171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.521589 (+ 418us) service_pool.cc:227] Handling call
0927 19:38:32.530826 (+ 9237us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:33.280304 (+749478us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.886495 (+606191us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b54004fddd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:33.886779 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1708222720}
I20260927 19:38:33.922681 3334 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:33.928542 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:33.961676 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 45734.9us
I20260927 19:38:33.997025 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 262) took 2474 ms. Trace:
I20260927 19:38:33.962729 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:33.996269 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1030) took 2080 ms. Trace:
I20260927 19:38:33.968801 2064 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:33.965207 2069 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:33.962095 2063 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:33.963089 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47320.3us
I20260927 19:38:33.951436 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.007807 2143 rpcz_store.cc:276] 0927 19:38:31.522840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.562704 (+ 39864us) service_pool.cc:227] Handling call
0927 19:38:31.562993 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:31.873757 (+310764us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:31.873786 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:31.873828 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:31.873865 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.996596 (+2122731us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400341a54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:33.996653 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:33.996847 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":714163072}
I20260927 19:38:33.986661 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.008333 1991 rpcz_store.cc:276] 0927 19:38:31.915583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.916480 (+ 897us) service_pool.cc:227] Handling call
0927 19:38:31.916763 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:32.100082 (+183319us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.995742 (+1895660us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54003039d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:33.996076 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":376241920}
I20260927 19:38:33.989311 2071 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
W20260927 19:38:33.967818 2068 consensus_meta.cc:287] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.560s user 0.011s sys 0.000s
I20260927 19:38:33.916910 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:33.966692 2067 consensus_meta.cc:287] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.574s user 0.009s sys 0.002s
I20260927 19:38:34.019788 3745 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.020076 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.020632 3745 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.024051 3745 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.026526 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:34.026798 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.029182 3833 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:34.029760 1991 raft_consensus.cc:2362] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4. Re-sending same reply.
I20260927 19:38:34.029884 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.030007 3833 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.031261 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.032826 3833 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.033320 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.033784 3833 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.028625 1993 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:33.961236 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:33.929070 2065 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:33.994580 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.038950 2071 raft_consensus.cc:1226] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:34.039237 2138 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:34.039399 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.040457 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:33.963212 1987 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 1031) took 2019 ms (client timeout 1585 ms). Trace:
I20260927 19:38:34.041884 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 1993) took 1417 ms. Trace:
I20260927 19:38:34.042430 2140 rpcz_store.cc:276] 0927 19:38:32.620710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.621181 (+ 471us) service_pool.cc:227] Handling call
0927 19:38:34.037757 (+1416576us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400076cd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:34.038009 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1384127360}
I20260927 19:38:34.041124 2068 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:33.968117 3977 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.050271 3655 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.045028 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:34.036403 1990 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.044474 2067 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:34.052069 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.055109 3908 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:34.057227 4037 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 1.391s user 0.009s sys 0.006s
I20260927 19:38:34.081300 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.082214 1988 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:34.062373 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.082872 2143 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:34.043792 3978 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.062786 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.058815 3655 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.061913 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.057834 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.071347 3912 consensus_queue.cc:1046] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260927 19:38:34.070891 1962 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:34.075587 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.078126 3334 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.063182 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.085662 3906 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:34.092027 3912 raft_consensus.cc:3032] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:34.092798 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:34.095485 2683 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34936 (request call id 570) took 1970 ms (client timeout 1500 ms). Trace:
I20260927 19:38:34.095803 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.096024 3852 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:34.096386 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 102077us
I20260927 19:38:34.100754 1994 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:34.107468 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.107023 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.108402 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:34.110042 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.111418 2681 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260927 19:38:34.112998 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1389) took 1875 ms. Trace:
I20260927 19:38:34.108891 2143 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:34.109031 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:34.119100 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 158235us
I20260927 19:38:34.119881 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 259) took 2111 ms. Trace:
I20260927 19:38:34.121532 3334 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.119175 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:34.123095 1988 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:34.123289 2688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 359) took 2440 ms. Trace:
I20260927 19:38:34.123971 3334 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.125833 1991 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.126421 4010 raft_consensus.cc:3032] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:34.126961 4010 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:34.128017 4010 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.128728 3969 consensus_queue.cc:1046] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:34.120359 1996 rpcz_store.cc:276] 0927 19:38:32.008296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:33.085902 (+1077606us) service_pool.cc:227] Handling call
0927 19:38:33.246866 (+160964us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:33.247060 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:33.247088 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:33.247136 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:33.247177 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.247333 (+ 156us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:34.119445 (+872112us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540031bd50. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:34.119664 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369507072}
I20260927 19:38:34.123740 2688 rpcz_store.cc:276] 0927 19:38:31.682288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.682425 (+ 137us) service_pool.cc:227] Handling call
0927 19:38:33.381764 (+1699339us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:33.381991 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:33.382019 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:33.382079 (+ 60us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:33.382124 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.382192 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:34.122849 (+740657us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b540061fdd0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:34.123065 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3907155200}
W20260927 19:38:34.043210 1987 rpcz_store.cc:269] 0927 19:38:31.944105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.947854 (+ 3749us) service_pool.cc:227] Handling call
0927 19:38:33.172870 (+1225016us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02bce 000000000034aea9 00007f791882ea30 00007f79188f47b4 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:33.949943 (+777073us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b540036f954. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:33.963128 (+ 13185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1020179072}
I20260927 19:38:34.102375 3912 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:34.106217 3978 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:34.140228 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.141927 2064 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 1.
I20260927 19:38:34.143786 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.129245 4010 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:34.103307 2683 rpcz_store.cc:269] 0927 19:38:32.125170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.095015 (+1969845us) service_pool.cc:214] Skipping call since client already timed out
0927 19:38:34.095239 (+ 224us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260927 19:38:34.146878 3829 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260927 19:38:34.103425 3833 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.149981 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.145501 2679 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:34.155659 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.156445 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.157575 2682 raft_consensus.cc:2396] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260927 19:38:34.157055 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:34.139779 3978 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:34.161237 2145 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:34.164624 3888 raft_consensus.cc:3032] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:34.149140 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.165117 3888 raft_consensus.cc:729] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:34.166695 2136 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:34.166296 3888 consensus_queue.cc:260] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.167850 3888 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.173885 3912 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.175405 3912 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.175983 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 1999) took 1066 ms. Trace:
I20260927 19:38:34.176388 2139 rpcz_store.cc:276] 0927 19:38:33.109969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:33.110210 (+ 241us) service_pool.cc:227] Handling call
0927 19:38:34.175487 (+1065277us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b54003822d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:34.175824 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68139776}
I20260927 19:38:34.178434 2111 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:34.113538 2066 rpcz_store.cc:276] 0927 19:38:32.237388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.262452 (+ 25064us) service_pool.cc:227] Handling call
0927 19:38:32.271906 (+ 9454us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:32.276165 (+ 4259us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:32.276207 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:32.303842 (+ 27635us) log.cc:842] Serialized 52 byte log entry
0927 19:38:32.573468 (+269626us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:32.573510 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:32.573657 (+ 147us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:34.056182 (+1482525us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:34.112316 (+ 56134us) raft_consensus.cc:1671] finished
0927 19:38:34.112338 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:34.112747 (+ 409us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10413,"raft.queue_time_us":127,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2567,"spinlock_wait_cycles":13673856,"thread_start_us":249590,"threads_started":1,"wal-append.queue_time_us":303830}
I20260927 19:38:34.179131 4034 consensus_queue.cc:1046] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:34.180344 4034 raft_consensus.cc:3032] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:34.180696 4034 raft_consensus.cc:729] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:34.182724 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 257) took 2204 ms. Trace:
I20260927 19:38:34.183171 1995 rpcz_store.cc:276] 0927 19:38:31.978442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.980225 (+ 1783us) service_pool.cc:227] Handling call
0927 19:38:33.323055 (+1342830us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:33.963983 (+640928us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:33.964303 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:33.964330 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:33.964371 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:33.964405 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:33.964476 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:34.182381 (+217905us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b540036f950. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:34.182510 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1479838464}
I20260927 19:38:34.187959 4020 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.188694 4020 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.191548 4020 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.193619 3906 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.193320 4034 consensus_queue.cc:260] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.194195 3906 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:34.194950 4034 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.196187 3978 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:34.219898 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.192793 4020 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260927 19:38:34.225167 1991 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:34.227540 1995 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:34.219730 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.204526 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.232848 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:34.236227 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 57922.3us
I20260927 19:38:34.241186 2142 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:34.244427 2139 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.245245 1991 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:34.251278 3965 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:34.251412 3432 consensus_queue.cc:1046] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20260927 19:38:34.254240 3432 raft_consensus.cc:3032] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:34.254629 3432 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:34.255525 3432 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.257784 3833 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.244030 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.228515 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.269511 2140 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:34.246156 3977 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.289844 3747 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:34.289629 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.290424 3962 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.266781 3432 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.290871 4043 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.278097 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.373785 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.377122 2140 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:34.352881 3747 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.456233 2144 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260927 19:38:34.452826 4043 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.457952 4021 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.471892 4021 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.480921 2681 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:34.481865 4021 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.482107 3924 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:34.284950 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.413823 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.513177 3655 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.514089 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:34.515066 3747 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.515429 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.472452 3962 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.516575 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:34.516981 3978 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.518580 3962 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.518956 3978 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.519553 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.519801 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:34.521713 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:34.563016 2687 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:34.523049 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.524390 1988 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:34.524904 2110 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:34.525709 4043 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:34.528861 2140 raft_consensus.cc:1261] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:34.529340 2139 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:34.531527 4024 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:34.533192 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 291089us
I20260927 19:38:34.537738 4048 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.541543 2653 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:34.521294 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:34.546203 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 317639us
I20260927 19:38:34.551208 3962 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.529244 1994 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
W20260927 19:38:34.581691 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 60644.4us
I20260927 19:38:34.577211 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:34.566769 1993 consensus_meta.cc:287] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.515s user 0.023s sys 0.008s
I20260927 19:38:34.573431 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.569490 1991 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:34.587249 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:34.588268 1993 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:34.577919 1987 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:34.567616 2683 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
W20260927 19:38:34.580876 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 277756us
I20260927 19:38:34.594606 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:34.597482 3834 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:34.615794 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:34.616654 2142 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 7
W20260927 19:38:34.598001 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 73171us
I20260927 19:38:34.621845 1961 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:34.613770 3834 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:34.626982 3834 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:34.627089 4024 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.629511 3776 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:34.630132 3776 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.631050 4034 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:34.631788 4034 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:34.632580 3776 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.613786 3838 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:34.633961 3838 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:34.634335 3838 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:34.635459 3908 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:34.636037 3908 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:34.638269 2062 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:34.639143 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1399) took 1403 ms. Trace:
I20260927 19:38:34.640089 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 1997) took 1744 ms. Trace:
I20260927 19:38:34.639599 2062 rpcz_store.cc:276] 0927 19:38:33.235786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:33.243112 (+ 7326us) service_pool.cc:227] Handling call
0927 19:38:34.609423 (+1366311us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800043598. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02bce 000000000034aea9 00007f791882ea30 00007f79188f47b4 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:34.637803 (+ 28380us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.638809 (+ 1006us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.639076 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24993152}
I20260927 19:38:34.640659 2141 rpcz_store.cc:276] 0927 19:38:32.895953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.896121 (+ 168us) service_pool.cc:227] Handling call
0927 19:38:32.942939 (+ 46818us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:33.211213 (+268274us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.639622 (+1428409us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b5400343354. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:34.639886 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615384960}
I20260927 19:38:34.646782 2111 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:34.652627 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.624958 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1394) took 1275 ms. Trace:
I20260927 19:38:34.655185 1995 rpcz_store.cc:276] 0927 19:38:33.349300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.184610 (+835310us) service_pool.cc:227] Handling call
0927 19:38:34.225887 (+ 41277us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.227980 (+ 2093us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.624641 (+396661us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b54000343d0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:34.624850 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93122432}
I20260927 19:38:34.653339 2139 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:34.615002 3912 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:34.734611 2110 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:34.751288 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 143527us
I20260927 19:38:34.753466 3879 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:34.754222 3879 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.756153 3879 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.757120 1995 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:34.605618 3962 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.757838 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1070) took 1115 ms. Trace:
I20260927 19:38:34.758175 1995 rpcz_store.cc:276] 0927 19:38:33.642279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.756636 (+1114357us) service_pool.cc:227] Handling call
0927 19:38:34.756966 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.757563 (+ 597us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.757761 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.759006 2687 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:34.759343 1995 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:34.759600 3962 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.601979 3978 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
I20260927 19:38:34.767673 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1404) took 1123 ms. Trace:
I20260927 19:38:34.772330 1995 rpcz_store.cc:276] 0927 19:38:33.644899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.758870 (+1113971us) service_pool.cc:227] Handling call
0927 19:38:34.759199 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.767205 (+ 8006us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.767593 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.764798 3834 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.769508 1988 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260927 19:38:34.778200 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 152089us
I20260927 19:38:34.653811 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:34.720883 3852 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:34.652534 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:34.781517 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 163073us
I20260927 19:38:34.784510 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1402) took 1408 ms. Trace:
I20260927 19:38:34.784955 1988 rpcz_store.cc:276] 0927 19:38:33.367475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.606799 (+1239324us) service_pool.cc:227] Handling call
0927 19:38:34.747157 (+140358us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:34.775985 (+ 28828us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.776222 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.774863 3833 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:34.594188 4048 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.789104 1991 pending_rounds.cc:76] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Aborting all ops after (but not including) 4
I20260927 19:38:34.793965 2653 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:34.633112 4034 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.785992 4020 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:34.637176 3908 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.793788 1991 pending_rounds.cc:97] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260927 19:38:34.797210 3776 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:34.820192 4004 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:34.820487 3776 raft_consensus.cc:686] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:34.821442 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 172697us
I20260927 19:38:34.824438 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.828390 1991 raft_consensus.cc:2866] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:34.834625 1993 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260927 19:38:34.836654 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:34.837590 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1409) took 1185 ms. Trace:
I20260927 19:38:34.837940 1988 rpcz_store.cc:276] 0927 19:38:33.646942 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.785724 (+1138782us) service_pool.cc:227] Handling call
0927 19:38:34.805907 (+ 20183us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.824992 (+ 19085us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:34.825023 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:34.831581 (+ 6558us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:34.831630 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.831691 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:34.831819 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.839429 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:34.840410 4058 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:34.841660 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 263994us
W20260927 19:38:34.830879 4010 consensus_meta.cc:287] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.683s user 0.007s sys 0.015s
I20260927 19:38:34.847254 2145 raft_consensus.cc:1261] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260927 19:38:34.845044 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 57281.7us
I20260927 19:38:34.847651 1962 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:34.839138 1988 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:34.854930 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1411) took 1207 ms. Trace:
W20260927 19:38:34.852011 3906 consensus_meta.cc:287] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.657s user 0.008s sys 0.004s
I20260927 19:38:34.855351 1988 rpcz_store.cc:276] 0927 19:38:33.647736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.838671 (+1190935us) service_pool.cc:227] Handling call
0927 19:38:34.838922 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.854684 (+ 15762us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.854875 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.857982 1988 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:34.788653 3852 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.861078 1961 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:34.866138 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1413) took 1210 ms. Trace:
I20260927 19:38:34.866585 1988 rpcz_store.cc:276] 0927 19:38:33.648474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.857547 (+1209073us) service_pool.cc:227] Handling call
0927 19:38:34.857831 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.858272 (+ 441us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.858405 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.846128 3962 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:34.822021 3833 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.875855 2111 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:34.872303 3922 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 19:38:34.877089 3917 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.877411 3922 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:34.878080 3922 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:34.877811 3917 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.879866 3917 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.880054 3838 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.829869 3776 consensus_queue.cc:237] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.846098 3747 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260927 19:38:34.882457 3834 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:34.882998 3747 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.883556 3747 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.884092 4013 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.884652 4013 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.885619 3747 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.886502 4013 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.887002 3887 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260927 19:38:34.855958 3906 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.877103 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.874338 3908 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:34.906333 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1421) took 1247 ms. Trace:
I20260927 19:38:34.906968 1988 rpcz_store.cc:276] 0927 19:38:33.651750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.867209 (+1215459us) service_pool.cc:227] Handling call
0927 19:38:34.867503 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.874979 (+ 7476us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:34.892716 (+ 17737us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:34.898419 (+ 5703us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:34.898465 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.898521 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:34.898631 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:34.891780 2683 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:34.875097 3830 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.875976 4024 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.892942 3962 raft_consensus.cc:686] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
W20260927 19:38:34.927063 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 394413us
I20260927 19:38:34.919448 2684 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:34.931296 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.898005 2138 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 19:38:34.940459 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 98106.1us
I20260927 19:38:34.910447 4010 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.911092 3735 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:34.935545 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1400) took 1575 ms. Trace:
I20260927 19:38:34.951050 1993 rpcz_store.cc:276] 0927 19:38:33.359714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.620076 (+1260362us) service_pool.cc:227] Handling call
0927 19:38:34.778696 (+158620us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:34.835079 (+ 56383us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.935228 (+100149us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x7b540038c054. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:34.935472 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100605184}
I20260927 19:38:34.945143 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.841130 4058 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:34.904174 3971 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20260927 19:38:34.958514 3838 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.950740 4048 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:34.963824 3889 consensus_queue.cc:1046] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260927 19:38:34.964773 3879 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:34.965256 3879 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:34.965436 3990 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:34.966269 3879 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.967537 3879 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:34.964808 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 258) took 2873 ms. Trace:
I20260927 19:38:34.969743 1989 rpcz_store.cc:276] 0927 19:38:31.999484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:33.085081 (+1085597us) service_pool.cc:227] Handling call
0927 19:38:33.085487 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:33.198636 (+113149us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.821789 (+1623153us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b5400302fd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:34.873085 (+ 51296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":72241536}
I20260927 19:38:34.964867 3981 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:34.976441 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:34.959646 2682 raft_consensus.cc:2396] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 19:38:34.982923 2037 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:34.980219 3906 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:34.935449 3978 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260927 19:38:34.940171 4024 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:34.926723 3830 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:34.987512 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:34.922199 4063 consensus_queue.cc:1046] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.159s
I20260927 19:38:34.949554 3969 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:34.994462 3906 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:34.995007 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
W20260927 19:38:35.001694 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 57673.3us
I20260927 19:38:34.979631 3735 raft_consensus.cc:686] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:34.998425 2143 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:35.003954 2144 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:35.010105 1962 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:35.011170 1993 raft_consensus.cc:1261] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 19:38:35.000818 4092 log.cc:924] Time spent T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260927 19:38:35.012749 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:34.982646 4010 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.011765 3906 raft_consensus.cc:2741] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260927 19:38:35.011575 4013 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.053946 4013 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.029347 3912 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:34.985100 3962 consensus_queue.cc:237] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.003955 4024 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.035056 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:35.113953 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48783.7us
I20260927 19:38:35.119223 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:35.055513 3833 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:35.013546 3969 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.061012 3852 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:35.128062 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 77093.6us
I20260927 19:38:35.129528 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:35.130201 3833 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.130779 2037 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:35.130887 3833 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.132712 4021 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:35.132984 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1395) took 1783 ms. Trace:
I20260927 19:38:35.133415 4021 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:35.133945 4021 raft_consensus.cc:686] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:35.133338 1996 rpcz_store.cc:276] 0927 19:38:33.349988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.209953 (+859965us) service_pool.cc:227] Handling call
0927 19:38:34.248692 (+ 38739us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:34.282327 (+ 33635us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:34.282374 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:34.283048 (+ 674us) log.cc:842] Serialized 26 byte log entry
0927 19:38:34.593109 (+310061us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:34.593145 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.593213 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:35.132795 (+539582us) raft_consensus.cc:1671] finished
0927 19:38:35.132813 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.132934 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":309653,"threads_started":1,"wal-append.queue_time_us":572922}
I20260927 19:38:35.135452 1996 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260927 19:38:35.111390 3912 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.136302 3912 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:35.137679 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 134949us
I20260927 19:38:35.139333 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:35.135017 4021 consensus_queue.cc:237] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.143121 4013 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.143467 3917 consensus_queue.cc:1046] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:35.144834 4010 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.144970 3735 consensus_queue.cc:237] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.148546 3977 raft_consensus.cc:3032] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:35.148970 3912 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:35.151098 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 308724us
I20260927 19:38:35.151438 3906 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:35.152175 3906 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:35.153410 3906 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.149029 3977 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:35.155977 3912 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.155498 3977 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.157390 3977 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:35.162045 3833 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.162195 3830 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.227962 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:35.229256 1996 raft_consensus.cc:2379] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:35.231272 2142 consensus_meta.cc:287] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.606s user 0.011s sys 0.001s
I20260927 19:38:35.231808 2142 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:35.233196 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:35.264519 1962 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:35.260912 2141 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260927 19:38:35.266105 2141 pending_rounds.cc:76] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 1
I20260927 19:38:35.266407 2141 pending_rounds.cc:97] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:38:35.266695 2141 raft_consensus.cc:2866] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:35.160981 3922 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.272498 2141 pending_rounds.cc:97] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 19:38:35.272933 2141 raft_consensus.cc:2866] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:35.270159 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:35.281052 2142 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:35.280824 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.285688 1962 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:35.272146 3889 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:35.289146 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:35.294693 1996 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:35.285338 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:35.296191 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:35.293884 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 465898us
I20260927 19:38:35.275960 3969 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.280161 4058 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.287112 3990 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:35.298970 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:35.304215 3990 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.314355 3990 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:35.297119 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:35.330199 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.328575 1993 raft_consensus.cc:1226] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:35.303120 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2018) took 1135 ms. Trace:
I20260927 19:38:35.302079 3889 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:35.307520 3908 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:35.345819 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:35.337685 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.311849 4061 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:35.349860 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 476191us
I20260927 19:38:35.350193 4061 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.352012 2111 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:35.353878 1990 consensus_meta.cc:287] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 1.256s user 0.014s sys 0.023s
I20260927 19:38:35.354269 1990 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:35.311040 3988 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:35.354809 1990 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60286 (request call id 1417) took 1704 ms (client timeout 1777 ms). Trace:
I20260927 19:38:35.341697 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:35.355403 1990 rpcz_store.cc:269] 0927 19:38:33.650138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:33.896077 (+245939us) service_pool.cc:227] Handling call
0927 19:38:35.354751 (+1458674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:35.333645 3838 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:35.356606 3838 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:35.357308 4084 log.cc:924] Time spent T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 1.113s user 0.006s sys 0.009s
I20260927 19:38:35.357599 2683 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:35.358783 2070 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1392) took 3017 ms. Trace:
I20260927 19:38:35.358990 3838 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:35.359133 2070 rpcz_store.cc:276] 0927 19:38:32.341491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:32.342096 (+ 605us) service_pool.cc:227] Handling call
0927 19:38:32.342430 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:32.342841 (+ 411us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:32.342881 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:32.343343 (+ 462us) log.cc:842] Serialized 26 byte log entry
0927 19:38:34.034273 (+1690930us) spinlock_profiling.cc:243] Waited 942 us on lock 0x7b18002d5c20. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e9411 00007f79139365cf
0927 19:38:34.084551 (+ 50278us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:34.085218 (+ 667us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.085438 (+ 220us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:35.356451 (+1271013us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:35.358440 (+ 1989us) raft_consensus.cc:1671] finished
0927 19:38:35.358458 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.358616 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3263,"spinlock_wait_cycles":43565312,"thread_start_us":47338,"threads_started":1,"wal-append.queue_time_us":1872513}
I20260927 19:38:35.360320 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.361585 3833 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:35.354439 4061 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.364048 3833 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:35.335646 2138 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:35.369059 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:35.370584 4061 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:35.372835 3838 consensus_queue.cc:1046] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260927 19:38:35.373883 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:35.374102 4043 raft_consensus.cc:3032] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:35.371219 4061 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:35.374703 4043 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:35.337512 2136 rpcz_store.cc:276] 0927 19:38:34.168189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.177546 (+ 9357us) service_pool.cc:227] Handling call
0927 19:38:35.276750 (+1099204us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:35.288880 (+ 12130us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:35.298188 (+ 9308us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:35.298221 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:35.298266 (+ 45us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:35.298308 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.298382 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.300487 (+ 2105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68087040}
I20260927 19:38:35.375363 4024 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.375033 4061 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:35.376209 4043 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.379029 4043 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:35.386322 4002 raft_consensus.cc:3032] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:35.387006 4002 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:35.388079 4002 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.389676 3924 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:35.390273 3924 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:35.397404 1988 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260927 19:38:35.398108 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1075) took 1172 ms. Trace:
I20260927 19:38:35.398442 1988 rpcz_store.cc:276] 0927 19:38:34.225424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.908420 (+682996us) service_pool.cc:227] Handling call
0927 19:38:34.908856 (+ 436us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:35.397131 (+488275us) spinlock_profiling.cc:243] Waited 67.7 ms on lock 0x7b5400136120. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138f2730 00007f7913964300 00007f7913965371 00007f7913961edf 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc
0927 19:38:35.397844 (+ 713us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.398049 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155626240}
I20260927 19:38:35.399981 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1088) took 1107 ms. Trace:
I20260927 19:38:35.400466 2140 rpcz_store.cc:276] 0927 19:38:34.330398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.381768 (+ 51370us) service_pool.cc:227] Handling call
0927 19:38:34.459314 (+ 77546us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.571176 (+111862us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.399689 (+828513us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b54000767d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:35.399911 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":159017216}
W20260927 19:38:35.405296 4088 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.574s user 0.001s sys 0.002s
I20260927 19:38:35.407421 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.408506 2680 raft_consensus.cc:2396] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260927 19:38:35.410523 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:35.420300 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 459562us
I20260927 19:38:35.399224 4061 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.469748 1959 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:35.489682 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:35.489981 1987 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1398) took 2134 ms. Trace:
I20260927 19:38:35.490459 1987 rpcz_store.cc:276] 0927 19:38:33.355964 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.606298 (+1250334us) service_pool.cc:227] Handling call
0927 19:38:34.787306 (+181008us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:34.846666 (+ 59360us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:34.846724 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:34.866067 (+ 19343us) log.cc:842] Serialized 26 byte log entry
0927 19:38:35.300528 (+434461us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:35.300570 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.300742 (+ 172us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:35.489665 (+188923us) raft_consensus.cc:1671] finished
0927 19:38:35.489693 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.489899 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36654,"raft.queue_time_us":24488,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":15092,"thread_start_us":394809,"threads_started":1,"wal-append.queue_time_us":599340}
I20260927 19:38:35.472299 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:35.493605 3888 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:35.482775 3922 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:35.497125 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:35.498011 1988 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:35.492383 4061 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.505378 2141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:35.506224 2141 raft_consensus.cc:595] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:35.506565 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1028) took 3526 ms. Trace:
I20260927 19:38:35.505707 3889 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.506942 1992 rpcz_store.cc:276] 0927 19:38:31.889320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:31.889494 (+ 174us) service_pool.cc:227] Handling call
0927 19:38:31.903014 (+ 13520us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:31.903314 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:31.903344 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:31.908145 (+ 4801us) log.cc:842] Serialized 26 byte log entry
0927 19:38:34.172285 (+2264140us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b180030cb80. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e9411 00007f79139365cf
0927 19:38:34.270585 (+ 98300us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:34.276803 (+ 6218us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.337199 (+ 60396us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:35.404786 (+1067587us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b24001957e8. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910892cf4 00007f7910890dfb 00007f791091f559 00007f791091f1e5 00007f791091e82d 00007f79135cd46e 00007f79135e4c0c 00007f79135e3f36 00007f79135e3b58 00007f79135ed551
0927 19:38:35.415487 (+ 10701us) raft_consensus.cc:1671] finished
0927 19:38:35.415514 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.415675 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4070,"raft.queue_time_us":11106,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":12758,"spinlock_wait_cycles":40115200,"thread_start_us":56483,"threads_started":1,"wal-append.queue_time_us":2807530}
I20260927 19:38:35.508991 1988 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:38:35.506589 2141 raft_consensus.cc:603] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:35.511649 3993 consensus_queue.cc:1046] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:35.512512 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:35.513202 3993 raft_consensus.cc:3032] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:35.513734 3993 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:35.513903 4061 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:35.514891 4061 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:35.514698 3993 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.516379 4002 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:35.517735 4061 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.522389 4024 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:35.532227 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 178889us
I20260927 19:38:35.516155 3993 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:35.465058 4020 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.009s sys 0.010s
I20260927 19:38:35.434923 3981 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:35.523015 4024 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:35.582504 4020 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.010s sys 0.010s
I20260927 19:38:35.592319 2111 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:35.558363 3432 consensus_meta.cc:287] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.181s user 0.020s sys 0.007s
I20260927 19:38:35.588712 4004 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:35.591535 3852 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.599684 4020 raft_consensus.cc:686] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:35.617751 1995 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1408) took 1971 ms. Trace:
I20260927 19:38:35.594528 4058 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.603835 2681 raft_consensus.cc:1226] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 1. Current term is 2. Ops: []
I20260927 19:38:35.625924 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:35.614568 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 483798us
I20260927 19:38:35.630524 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:35.632783 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:35.633545 2144 raft_consensus.cc:595] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:35.611007 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.620919 3917 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:35.649581 2037 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:35.623824 4004 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.649298 4021 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:35.651839 4021 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.652904 4004 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.634722 4058 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.639686 3833 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:35.659533 2682 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:35.660812 1991 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1403) took 2285 ms. Trace:
I20260927 19:38:35.661707 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:35.661840 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.662425 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:35.629902 3852 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.664304 4058 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.665719 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:35.682217 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 211079us
I20260927 19:38:35.649535 3830 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:35.686201 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1399) took 2328 ms. Trace:
I20260927 19:38:35.662537 4021 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.683594 1991 rpcz_store.cc:276] 0927 19:38:33.375552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.608203 (+1232651us) service_pool.cc:227] Handling call
0927 19:38:34.750917 (+142714us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:34.896335 (+145418us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:34.896382 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:34.903070 (+ 6688us) log.cc:842] Serialized 52 byte log entry
0927 19:38:35.435386 (+532316us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:35.435421 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.435486 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:35.660496 (+225010us) raft_consensus.cc:1671] finished
0927 19:38:35.660529 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.660736 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29631,"raft.queue_time_us":30393,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":1465,"thread_start_us":488425,"threads_started":1,"wal-append.queue_time_us":718835}
I20260927 19:38:35.693128 1959 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:35.694324 2144 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:35.697906 1962 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:35.698910 1994 rpcz_store.cc:276] 0927 19:38:33.357656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.607910 (+1250254us) service_pool.cc:227] Handling call
0927 19:38:34.647890 (+ 39980us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.799255 (+151365us) raft_consensus.cc:1520] Early marking committed up to index 2
0927 19:38:35.268674 (+469419us) log.cc:842] Serialized 14 byte log entry
0927 19:38:35.685851 (+417177us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:35.685905 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:35.685958 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.686021 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.686148 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":694083,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":3593,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":808175,"threads_started":2}
W20260927 19:38:35.707253 3735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.002s sys 0.028s
W20260927 19:38:35.707927 3735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.003s sys 0.028s
I20260927 19:38:35.708895 1989 raft_consensus.cc:1226] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
W20260927 19:38:35.709857 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 48220.7us
I20260927 19:38:35.722216 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:35.723980 2144 raft_consensus.cc:595] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:35.724468 2144 raft_consensus.cc:603] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 19:38:35.725296 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 311197us
I20260927 19:38:35.690533 4105 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:38:35.728281 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 81616.3us
I20260927 19:38:35.625036 1995 rpcz_store.cc:276] 0927 19:38:33.646583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.773289 (+1126706us) service_pool.cc:227] Handling call
0927 19:38:34.789639 (+ 16350us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:34.846238 (+ 56599us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 19:38:35.306959 (+460721us) log.cc:842] Serialized 14 byte log entry
0927 19:38:35.617424 (+310465us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:35.617471 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:35.617531 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.617594 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:35.617700 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":647430,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":2901,"thread_start_us":744017,"threads_started":2}
I20260927 19:38:35.631170 3981 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.702630 4004 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:35.704526 3829 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:35.739298 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:35.723693 4013 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:35.729247 2680 raft_consensus.cc:2379] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:35.723310 4058 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:35.694325 3988 consensus_queue.cc:1046] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260927 19:38:35.733749 4020 consensus_queue.cc:237] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:35.759626 4108 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260927 19:38:35.760902 4106 raft_consensus.cc:2741] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260927 19:38:35.741638 4024 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.765960 4013 raft_consensus.cc:686] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:35.769951 3432 consensus_queue.cc:1046] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:35.771273 4106 consensus_queue.cc:1046] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:35.772514 4098 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:35.773500 3776 raft_consensus.cc:3032] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:35.772598 4106 consensus_queue.cc:1046] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:35.773897 3776 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:35.774681 4101 raft_consensus.cc:3032] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:35.775195 4101 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:35.776279 4101 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.777449 4101 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:35.774724 3990 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.738284 3830 raft_consensus.cc:686] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:35.743593 3829 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:35.778363 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.736186 3735 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:35.778261 4021 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:35.774818 3776 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.799345 3990 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.799818 3776 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:35.800048 1990 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:35.804056 3924 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.810621 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:35.809751 3924 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.747900 4004 raft_consensus.cc:686] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:35.814381 4004 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.820060 1994 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:35.823094 4043 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:35.825440 3829 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:35.827018 2684 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260927 19:38:35.793527 4058 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.834033 3917 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:35.834621 3917 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:35.835410 4058 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:35.836062 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 53627.4us
I20260927 19:38:35.837535 2108 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:35.838299 4107 raft_consensus.cc:3032] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:35.841380 4107 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:35.842247 4107 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.843727 4107 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:35.840659 4043 consensus_queue.cc:1046] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260927 19:38:35.850944 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 65773.8us
W20260927 19:38:35.856971 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 103348us
I20260927 19:38:35.773681 3432 raft_consensus.cc:3032] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:35.866431 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:35.871634 4021 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:35.873031 4021 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.880833 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0f797e07d634f6abe29556a11863581"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:35.881443 2145 raft_consensus.cc:595] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:35.882725 4109 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:35.883287 4109 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.858747 3833 raft_consensus.cc:2741] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260927 19:38:35.865566 4106 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:35.862429 3852 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:35.898377 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1077) took 1628 ms. Trace:
I20260927 19:38:35.862741 3432 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:35.864719 2138 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:35.905346 1989 rpcz_store.cc:276] 0927 19:38:34.270316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.970629 (+700313us) service_pool.cc:227] Handling call
0927 19:38:34.970933 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:35.709505 (+738572us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:35.898121 (+188616us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54001ee1d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:35.898300 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1696448896}
I20260927 19:38:35.861516 4102 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260927 19:38:35.923144 3962 raft_consensus.cc:3032] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:35.923563 3962 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:35.925681 3432 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.926832 3432 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:35.866902 4058 consensus_queue.cc:1046] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20260927 19:38:35.892589 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.895267 4109 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:35.886953 4024 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260927 19:38:35.891254 3906 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260927 19:38:35.901405 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:35.924427 3962 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.942925 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:35.943885 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:35.963891 2062 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:35.945225 2066 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:35.946573 2071 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:35.860610 3735 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.960804 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 234) took 9746 ms. Trace:
I20260927 19:38:35.978871 4098 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:35.979561 4098 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:35.981448 2064 raft_consensus.cc:1226] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:35.981735 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:35.982368 2067 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:35.982441 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:35.982935 2071 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:35.983107 2064 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:35.983332 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:35.983814 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:35.983945 2067 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:35.983989 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:35.984382 2071 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:35.961217 4132 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:35.961661 4089 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 1.180s user 0.001s sys 0.014s
I20260927 19:38:35.962111 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:35.962450 4115 log.cc:924] Time spent T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260927 19:38:35.963488 2070 raft_consensus.cc:2396] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:35.935683 4034 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:35.913424 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:35.907868 4106 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.985090 2067 raft_consensus.cc:1261] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:35.948202 4102 consensus_queue.cc:1225] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer 4e0335d1a85f421380029b95a91bf0c4 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 19:38:35.989426 2064 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:38:35.968560 2137 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:35.978185 1893 rpcz_store.cc:276] 0927 19:38:26.182732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:26.182925 (+ 193us) service_pool.cc:227] Handling call
0927 19:38:35.929245 (+9746320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:31.048454 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:31.049100 (+ 646us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:31.049139 (+ 39us) write_op.cc:435] Acquired schema lock
0927 19:38:31.049153 (+ 14us) tablet.cc:660] Decoding operations
0927 19:38:31.049599 (+ 446us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:31.049657 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:31.049695 (+ 38us) tablet.cc:683] Acquiring locks for 2 operations
0927 19:38:31.119455 (+ 69760us) tablet.cc:699] Row locks acquired
0927 19:38:31.119471 (+ 16us) write_op.cc:260] PREPARE: finished
0927 19:38:31.132553 (+ 13082us) write_op.cc:270] Start()
0927 19:38:31.138994 (+ 6441us) write_op.cc:276] Timestamp: P: 1790537911128659 usec, L: 0
0927 19:38:31.139020 (+ 26us) op_driver.cc:331] REPLICATION: starting
0927 19:38:31.296924 (+157904us) log.cc:842] Serialized 929 byte log entry
0927 19:38:32.215075 (+918151us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:32.623278 (+408203us) op_driver.cc:444] REPLICATION: finished
0927 19:38:32.888408 (+265130us) write_op.cc:301] APPLY: starting
0927 19:38:32.888491 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:32.888621 (+ 130us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:32.888647 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:32.888980 (+ 333us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:32.889151 (+ 171us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0927 19:38:32.889207 (+ 56us) write_op.cc:312] APPLY: finished
0927 19:38:32.889463 (+ 256us) log.cc:842] Serialized 29 byte log entry
0927 19:38:32.889653 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:32.889747 (+ 94us) write_op.cc:454] Released schema lock
0927 19:38:32.889793 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:32.890021 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:32.890051 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":239644,"apply.run_cpu_time_us":1917,"apply.run_wall_time_us":1916,"num_ops":2,"prepare.queue_time_us":984725,"prepare.run_cpu_time_us":24693,"prepare.run_wall_time_us":476211,"raft.queue_time_us":236439,"raft.run_cpu_time_us":24489,"raft.run_wall_time_us":226815,"replication_time_us":1454885,"spinlock_wait_cycles":8173952,"thread_start_us":983556,"threads_started":4,"wal-append.queue_time_us":419825,"wal-append.run_cpu_time_us":28677,"wal-append.run_wall_time_us":2172168}]]}
I20260927 19:38:35.965101 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:36.013509 2141 raft_consensus.cc:1226] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260927 19:38:36.034231 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:36.016585 3971 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:35.974301 4013 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.018225 4021 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:36.045616 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:36.025507 2138 raft_consensus.cc:595] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:36.042270 2143 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:36.043403 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:36.044538 1996 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:36.035593 2065 raft_consensus.cc:2379] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:36.045239 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.053215 2064 raft_consensus.cc:2396] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:35.950873 3830 consensus_queue.cc:237] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:35.987952 4132 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:36.054569 1996 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:38:36.003758 4034 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.060675 2139 raft_consensus.cc:1226] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:36.062024 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:36.063053 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.055291 2688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 594) took 1896 ms. Trace:
I20260927 19:38:36.065063 1990 raft_consensus.cc:1226] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:36.063769 2067 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:36.067133 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.048413 4024 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:36.073164 4128 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260927 19:38:36.051235 2138 raft_consensus.cc:603] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:36.072718 4024 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.071413 2066 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:36.064545 2688 rpcz_store.cc:276] 0927 19:38:34.159057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:34.159194 (+ 137us) service_pool.cc:227] Handling call
0927 19:38:34.171879 (+ 12685us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:34.172224 (+ 345us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:34.172253 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:34.172701 (+ 448us) log.cc:842] Serialized 26 byte log entry
0927 19:38:34.218288 (+ 45587us) raft_consensus.cc:1631] Marking committed up to 7
0927 19:38:34.218343 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:34.218442 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:35.704696 (+1486254us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:36.049470 (+344774us) raft_consensus.cc:1671] finished
0927 19:38:36.049489 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:36.055232 (+ 5743us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14912,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"spinlock_wait_cycles":47940480,"thread_start_us":45138,"threads_started":1,"wal-append.queue_time_us":580983}
I20260927 19:38:36.056480 4058 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:36.096426 2140 raft_consensus.cc:1226] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:36.097229 1992 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260927 19:38:36.098368 2139 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260927 19:38:36.093164 4125 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20260927 19:38:36.171645 4126 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260927 19:38:36.171381 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.081887 4098 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:36.172634 2654 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.131:45045 timed out after 1.993s (SENT) [suppressed 31 similar messages]
W20260927 19:38:36.174846 4116 log.cc:924] Time spent T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.246s user 0.000s sys 0.001s
W20260927 19:38:36.176087 4129 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260927 19:38:36.176172 3879 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:36.178085 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 72898.3us
I20260927 19:38:36.176786 3879 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.214030 4004 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:36.222244 2066 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: [3.3-4.4]
W20260927 19:38:36.224684 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 52075.6us
I20260927 19:38:36.071084 4021 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:36.235849 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.240396 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:36.242362 2066 raft_consensus.cc:2396] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:36.227385 4105 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:36.049119 3971 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.277098 4105 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.265142 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.180716 4058 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.257601 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260927 19:38:36.269572 3993 consensus_meta.cc:287] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.678s user 0.009s sys 0.003s
I20260927 19:38:36.272670 3735 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:36.282013 2067 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:36.242779 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.284273 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:36.284860 3977 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:36.289702 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:36.291226 3917 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.294269 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.295038 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.297739 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.298687 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.308135 2143 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:36.310367 2064 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:36.301317 1993 raft_consensus.cc:1261] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:36.304997 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:36.306141 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.319031 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:36.336820 1990 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:36.342672 2686 raft_consensus.cc:3032] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Stepping down as leader of term 3
W20260927 19:38:36.343146 4133 log.cc:924] Time spent T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260927 19:38:36.343201 2686 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
W20260927 19:38:36.311959 4131 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.275s user 0.000s sys 0.002s
I20260927 19:38:36.344620 2066 raft_consensus.cc:1226] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:36.344266 2686 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:36.345611 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:36.345728 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:36.346783 4107 consensus_meta.cc:287] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.503s user 0.006s sys 0.004s
I20260927 19:38:36.346863 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:36.347738 2140 raft_consensus.cc:3032] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:38:36.342519 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:36.345306 2686 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:36.348060 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.317159 2141 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:36.348124 2140 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:36.354255 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:38:36.374362 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 196309us
I20260927 19:38:36.319752 2679 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:36.593909 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.440887 4002 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:36.502312 3917 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:36.599372 4140 log.cc:924] Time spent T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260927 19:38:36.425738 2068 raft_consensus.cc:1226] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:36.449200 4034 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:36.600399 4105 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:36.404042 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.859735 2139 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 7: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:36.360553 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:36.359524 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:36.503391 4147 log.cc:924] Time spent T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20260927 19:38:36.842829 3906 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:36.342751 3879 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:36.902998 1959 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:36.404342 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:36.904393 2066 raft_consensus.cc:2379] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:36.488175 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 85994.8us
W20260927 19:38:36.433297 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 124314us
I20260927 19:38:36.906657 2111 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:36.911269 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.510830 4024 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:36.314419 3977 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:36.921787 4138 log.cc:924] Time spent T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.669s user 0.000s sys 0.001s
I20260927 19:38:36.921934 3977 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:36.348793 3829 consensus_meta.cc:287] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.523s user 0.005s sys 0.005s
I20260927 19:38:36.923434 3829 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.924203 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.348527 2138 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:36.349150 4106 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:36.925789 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2056) took 1025 ms. Trace:
I20260927 19:38:36.926306 2111 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:36.926680 3981 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:36.855403 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:36.927708 3432 consensus_meta.cc:287] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.000s user 0.006s sys 0.004s
I20260927 19:38:36.928251 3432 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:36.928671 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:36.595202 3971 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:36.925391 3829 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:36.902390 3988 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:36.857249 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.931260 2069 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:36.930892 3988 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.929199 3981 consensus_queue.cc:1046] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260927 19:38:36.932538 2069 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260927 19:38:36.933777 3981 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:36.934630 4106 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:36.934763 3981 raft_consensus.cc:3032] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:36.935122 4106 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:36.935206 3981 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:36.935626 4106 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:36.937232 3981 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.926121 2145 rpcz_store.cc:276] 0927 19:38:35.900931 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:35.901099 (+ 168us) service_pool.cc:227] Handling call
0927 19:38:35.912374 (+ 11275us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:35.962364 (+ 49990us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:35.962395 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:35.962856 (+ 461us) log.cc:842] Serialized 26 byte log entry
0927 19:38:36.025393 (+ 62537us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:36.046915 (+ 21522us) log.cc:842] Serialized 14 byte log entry
0927 19:38:36.051536 (+ 4621us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:36.051683 (+ 147us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:36.925575 (+873892us) raft_consensus.cc:1671] finished
0927 19:38:36.925612 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:36.925740 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29664,"prepare.queue_time_us":32844,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":34429,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":17762,"spinlock_wait_cycles":7881600,"thread_start_us":62127,"threads_started":1,"wal-append.queue_time_us":264994}
I20260927 19:38:36.941990 4058 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:36.930074 4101 consensus_meta.cc:287] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.152s user 0.007s sys 0.004s
W20260927 19:38:36.945636 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 55046.8us
W20260927 19:38:36.504383 4144 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.204s user 0.000s sys 0.001s
I20260927 19:38:36.503499 4021 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:36.842931 4142 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:36.943789 3981 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:36.947562 4142 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:36.949460 4106 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.925292 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:36.950400 4142 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:36.958660 2686 consensus_meta.cc:287] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.603s user 0.009s sys 0.000s
I20260927 19:38:36.959411 2686 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260927 19:38:36.599592 4103 log.cc:924] Time spent T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260927 19:38:36.952153 3977 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:36.953478 3906 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:36.953795 3971 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:36.955211 2062 consensus_meta.cc:287] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.991s user 0.009s sys 0.000s
I20260927 19:38:36.953168 2066 raft_consensus.cc:1261] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260927 19:38:36.966734 2066 pending_rounds.cc:76] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 1
I20260927 19:38:36.967108 2066 pending_rounds.cc:97] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:38:36.967509 2066 raft_consensus.cc:2866] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:36.962958 3977 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:36.968761 3977 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:36.973786 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:36.974293 2686 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:36.502374 4095 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:36.960614 2070 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
W20260927 19:38:36.860114 2137 consensus_meta.cc:287] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.826s user 0.004s sys 0.005s
I20260927 19:38:36.974546 2679 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:36.974890 3906 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:36.961936 1992 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.015s sys 0.009s
I20260927 19:38:36.975096 3971 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:36.772004 4048 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:36.448421 3993 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:36.933178 3988 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:36.424728 4061 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.627s
I20260927 19:38:36.961957 4021 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:37.157619 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 231367us
W20260927 19:38:37.471815 2034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.000s sys 0.014s
W20260927 19:38:37.472221 2034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.000s sys 0.014s
W20260927 19:38:37.060498 2110 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.002s sys 0.027s
W20260927 19:38:37.464893 1961 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.001s sys 0.018s
W20260927 19:38:37.465250 2654 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.011s sys 0.020s
W20260927 19:38:37.474828 2654 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.012s sys 0.020s
I20260927 19:38:37.475533 2071 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260927 19:38:37.475916 2071 pending_rounds.cc:76] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 2
I20260927 19:38:37.477612 2071 pending_rounds.cc:97] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260927 19:38:36.589950 3852 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:37.060325 4013 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.004s sys 0.025s
I20260927 19:38:37.468163 4024 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:37.055171 2684 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.003s sys 0.027s
W20260927 19:38:37.086226 4043 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.002s sys 0.029s
W20260927 19:38:37.144815 4152 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260927 19:38:37.201699 4020 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:37.035822 2109 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.005s sys 0.007s
W20260927 19:38:37.493724 2109 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.145s user 0.009s sys 0.013s
W20260927 19:38:37.470867 1992 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.171s user 0.023s sys 0.018s
W20260927 19:38:37.474531 2110 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.119s user 0.003s sys 0.037s
I20260927 19:38:36.502699 4155 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:37.478161 1961 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.527s user 0.002s sys 0.018s
I20260927 19:38:37.480171 2071 raft_consensus.cc:2866] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 19:38:37.479966 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.004s sys 0.015s
I20260927 19:38:37.457140 4098 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:37.148192 3977 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.055964 2685 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.492221 2654 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:37.496752 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:37.498395 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.005s sys 0.015s
I20260927 19:38:36.504369 4109 raft_consensus.cc:3032] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:36.988044 2062 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:37.517323 2067 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:37.500912 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1436) took 1202 ms. Trace:
W20260927 19:38:37.484853 4013 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.168s user 0.011s sys 0.030s
I20260927 19:38:37.502447 4155 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.488399 4048 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:37.503774 4148 log.cc:924] Time spent T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.336s user 0.000s sys 0.000s
W20260927 19:38:37.504674 4156 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.311s user 0.000s sys 0.000s
W20260927 19:38:37.489982 2684 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.981s user 0.007s sys 0.034s
I20260927 19:38:37.014810 2137 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:37.488571 3981 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260927 19:38:37.511956 3833 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:37.498049 4043 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.145s user 0.005s sys 0.036s
I20260927 19:38:37.536865 3833 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.500118 4106 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:37.537731 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.538760 4106 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:37.538892 3833 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:37.539347 4106 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.540731 3962 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:37.487540 4021 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.543005 4021 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:37.512671 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1428) took 1481 ms. Trace:
I20260927 19:38:37.493222 3906 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:37.544684 3906 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.516942 4109 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:37.489795 3852 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.543720 1988 rpcz_store.cc:276] 0927 19:38:36.031358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.031503 (+ 145us) service_pool.cc:227] Handling call
0927 19:38:36.031867 (+ 364us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:36.038707 (+ 6840us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:36.038726 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:36.056532 (+ 17806us) log.cc:842] Serialized 26 byte log entry
0927 19:38:36.441809 (+385277us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:36.823872 (+382063us) log.cc:842] Serialized 14 byte log entry
0927 19:38:36.886394 (+ 62522us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:36.886507 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:37.512458 (+625951us) raft_consensus.cc:1671] finished
0927 19:38:37.512478 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:37.512611 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":381849,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":76064,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":51476,"thread_start_us":262120,"threads_started":1,"wal-append.queue_time_us":981903}
I20260927 19:38:37.036690 3978 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:37.126426 1891 catalog_manager.cc:5495] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 5, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:37.518404 1992 rpcz_store.cc:276] 0927 19:38:36.298169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.298312 (+ 143us) service_pool.cc:227] Handling call
0927 19:38:36.298640 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.298840 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 19:38:36.299140 (+ 300us) log.cc:842] Serialized 14 byte log entry
0927 19:38:37.500607 (+1201467us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:37.500654 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:37.500693 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:37.500748 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:37.500860 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":444,"prepare.queue_time_us":3683,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":39,"thread_start_us":1195874,"threads_started":1}
I20260927 19:38:36.854493 4002 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.526409 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.557528 2071 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:37.500150 3988 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260927 19:38:37.537010 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.501289 4020 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.549127 4161 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:37.566483 4161 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.567399 4002 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:37.567586 3852 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:37.567648 2143 raft_consensus.cc:1226] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:37.547829 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1433) took 1503 ms. Trace:
I20260927 19:38:37.568533 1993 rpcz_store.cc:276] 0927 19:38:36.044182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.044317 (+ 135us) service_pool.cc:227] Handling call
0927 19:38:36.063166 (+ 18849us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.312362 (+249196us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:37.538307 (+1225945us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400194454. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:37.547755 (+ 9448us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":546946816}
W20260927 19:38:37.551831 2137 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40164 (request call id 2052) took 1703 ms (client timeout 1882 ms). Trace:
W20260927 19:38:37.569689 2137 rpcz_store.cc:269] 0927 19:38:35.845815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:35.854653 (+ 8838us) service_pool.cc:227] Handling call
0927 19:38:37.549150 (+1694497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:37.570386 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1851) took 1267 ms. Trace:
I20260927 19:38:37.570935 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.571424 4048 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:37.571851 2137 raft_consensus.cc:2354] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:37.567440 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.568893 4161 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:37.544391 2140 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.534929 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.577970 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.562816 2145 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:37.562887 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.580116 2071 raft_consensus.cc:2379] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:37.488508 3993 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.566906 4020 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:37.531991 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1410) took 1579 ms. Trace:
I20260927 19:38:37.498253 4024 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.553472 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:37.555356 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.517848 4098 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.570775 2067 rpcz_store.cc:276] 0927 19:38:36.303095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.303222 (+ 127us) service_pool.cc:227] Handling call
0927 19:38:37.517107 (+1213885us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:37.520500 (+ 3393us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:37.570166 (+ 49666us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400322b50. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:37.570320 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2728892160}
I20260927 19:38:37.582877 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.585230 2140 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:37.569090 4002 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:37.588675 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.592236 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.591733 1891 catalog_manager.cc:5495] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:37.600104 2066 raft_consensus.cc:2396] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:37.562217 3971 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:37.620906 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 254519us
I20260927 19:38:37.644369 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.647359 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:37.648192 2067 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:37.648948 2034 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:37.651769 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:37.564853 3978 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.660609 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.602041 4173 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:37.666446 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.622148 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.673318 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.679330 2071 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:37.587826 2062 rpcz_store.cc:276] 0927 19:38:35.942843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:35.943535 (+ 692us) service_pool.cc:227] Handling call
0927 19:38:37.521966 (+1578431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:37.665525 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:37.665014 1891 catalog_manager.cc:5495] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 2, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:37.644450 3971 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:37.611248 4168 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:37.681572 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 156817us
I20260927 19:38:37.682639 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.686846 2071 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 1.
I20260927 19:38:37.690654 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:37.669387 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 177167us
I20260927 19:38:37.695117 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.699537 2071 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:37.694371 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:37.665009 4106 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:37.702538 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.706570 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:37.703294 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:37.707674 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.708297 1996 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:37.708159 2686 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:37.711826 2144 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:37.713991 1995 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:37.714673 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.711609 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.718171 1991 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:37.719708 2141 raft_consensus.cc:2396] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260927 19:38:37.720214 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.720475 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.720868 2071 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:37.721038 1993 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:37.639390 4002 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.724505 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:37.724785 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1863) took 1281 ms. Trace:
I20260927 19:38:37.724274 1992 raft_consensus.cc:1226] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:37.712212 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.672947 4174 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:37.728621 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.729393 2071 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:37.736109 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.695170 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.743731 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.659741 4109 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.707434 4020 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:37.676712 3833 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260927 19:38:37.753288 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.753125 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.747052 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:37.759630 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.761968 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:37.762708 2071 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:37.683535 4173 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:37.766822 4158 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20260927 19:38:37.767205 4173 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:37.703830 3971 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.769138 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:37.769829 2684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 613) took 1509 ms. Trace:
I20260927 19:38:37.770292 2684 rpcz_store.cc:276] 0927 19:38:36.260654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.268771 (+ 8117us) service_pool.cc:227] Handling call
0927 19:38:36.507856 (+239085us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:36.508161 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.508481 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 8
0927 19:38:37.538785 (+1030304us) log.cc:842] Serialized 14 byte log entry
0927 19:38:37.769448 (+230663us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:37.769510 (+ 62us) raft_consensus.cc:1631] Marking committed up to 8
0927 19:38:37.769558 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:37.769634 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:37.769764 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":464696,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":4201,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":36676352,"thread_start_us":1238691,"threads_started":2}
I20260927 19:38:37.761973 4155 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:37.773236 4166 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:37.736055 4174 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:37.735373 4095 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:37.745606 4161 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:37.776779 2139 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:37.780323 2140 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260927 19:38:37.763844 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.792884 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.811708 1992 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:37.793395 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.797091 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.797804 2067 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:37.813023 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.814402 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.815047 2067 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:37.799428 1996 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:37.745608 3977 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:37.811060 2140 pending_rounds.cc:76] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 6
I20260927 19:38:37.818055 2140 pending_rounds.cc:97] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260927 19:38:37.818403 2140 raft_consensus.cc:2866] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:37.819137 2140 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46930 (request call id 294) took 1521 ms. Trace:
I20260927 19:38:37.819432 2140 rpcz_store.cc:276] 0927 19:38:36.297511 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.347369 (+ 49858us) service_pool.cc:227] Handling call
0927 19:38:36.347616 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:37.543838 (+1196222us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b4c001ada00. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7911d25210 00007f7911d35fb2 00007f7911cfcac5 00007f7911d2371d 00007f7911d23383 00007f791395d4fb 00007f791395e025 00007f79139592ef 00007f791395ab4e 00007f7913964c89 00007f7913965556
0927 19:38:37.818936 (+275098us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:37.819091 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16718592}
I20260927 19:38:37.821659 1996 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260927 19:38:37.822011 2067 pending_rounds.cc:76] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 1
I20260927 19:38:37.822317 2067 pending_rounds.cc:97] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:38:37.824862 2067 raft_consensus.cc:2866] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:37.844869 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.854566 1991 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:38:37.759747 4102 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:37.854430 1992 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:37.761791 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:37.811131 4109 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:37.868429 1990 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260927 19:38:37.869474 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 204026us
I20260927 19:38:37.811008 4161 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:37.762013 4020 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.795819 4174 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:37.862989 4098 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:37.857533 3908 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260927 19:38:37.868340 4024 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:37.812422 4095 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:37.870677 2062 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:37.882584 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.872825 2110 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:37.877188 4161 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:37.881902 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:37.876605 4169 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:37.863263 4102 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:38.023766 2144 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260927 19:38:38.110780 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1425) took 2131 ms. Trace:
W20260927 19:38:37.881096 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 215342us
I20260927 19:38:37.955816 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:38.131032 2063 rpcz_store.cc:276] 0927 19:38:35.979257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:35.979620 (+ 363us) service_pool.cc:227] Handling call
0927 19:38:35.979949 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.092468 (+112519us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:36.092497 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:36.092540 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:36.092569 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.110536 (+2017967us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400323054. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:38.110587 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:38.110718 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234882432}
W20260927 19:38:37.966782 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 242748us
I20260927 19:38:38.024142 2062 raft_consensus.cc:3032] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:38.133841 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:38.134620 2062 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:37.924439 1993 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:37.923202 2142 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:38.144963 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.010741 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.148895 1995 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:37.880997 3906 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:37.885074 4179 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.148584 1962 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:38.151583 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 278849us
I20260927 19:38:38.153643 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.154241 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:38.151881 3962 consensus_meta.cc:287] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.611s user 0.012s sys 0.000s
W20260927 19:38:38.043152 4177 log.cc:924] Time spent T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.178s user 0.000s sys 0.001s
I20260927 19:38:38.151929 4169 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:38.158543 4178 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.193s user 0.000s sys 0.000s
I20260927 19:38:38.156281 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.157593 1995 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:38.156841 2062 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:38.163501 2062 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:38.166316 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.146876 4183 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.167655 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.029270 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.082883 3971 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:38.009915 4043 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.147559 3993 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.170820 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.030889 4002 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.156602 4155 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260927 19:38:38.239283 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 136134us
W20260927 19:38:38.333474 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 159741us
I20260927 19:38:38.350565 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:38.357052 1961 leader_election.cc:334] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.669s (SENT)
I20260927 19:38:38.333271 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.177748 2110 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:38.267565 2071 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:38.282943 1995 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:38.353652 1993 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:38.269613 3833 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.368396 3833 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:38.370445 3833 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.372128 3833 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:38.372355 2037 connection.cc:232] RPC call timeout handler was delayed by -3.1032239070000287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260927 19:38:38.372900 2037 connection.cc:232] RPC call timeout handler was delayed by -3.1020930500001214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20260927 19:38:38.373262 2037 connection.cc:232] RPC call timeout handler was delayed by -3.0209583360001488s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.254s for a response
W20260927 19:38:38.373627 2037 connection.cc:232] RPC call timeout handler was delayed by -3.0002583360001154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25630000000000003s for a response
W20260927 19:38:38.374161 2037 connection.cc:232] RPC call timeout handler was delayed by -2.9638572410000279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260927 19:38:38.374756 2037 connection.cc:232] RPC call timeout handler was delayed by -2.7803817070000605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2932s for a response
W20260927 19:38:38.375202 2037 connection.cc:232] RPC call timeout handler was delayed by -2.7275944150001123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20260927 19:38:38.375573 2037 connection.cc:232] RPC call timeout handler was delayed by -2.6823035060000393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
W20260927 19:38:38.375922 2037 connection.cc:232] RPC call timeout handler was delayed by -2.4557944150001276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18380000000000002s for a response
W20260927 19:38:38.376273 2037 connection.cc:232] RPC call timeout handler was delayed by -2.3792675220000774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3331s for a response
W20260927 19:38:38.376621 2037 connection.cc:232] RPC call timeout handler was delayed by -2.3315384160000576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
W20260927 19:38:38.376995 2037 connection.cc:232] RPC call timeout handler was delayed by -2.2503379250000535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260927 19:38:38.377519 3906 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.378175 3906 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.380316 3906 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.381016 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1875) took 1221 ms. Trace:
I20260927 19:38:38.381449 2067 rpcz_store.cc:276] 0927 19:38:37.159713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:37.821545 (+661832us) service_pool.cc:227] Handling call
0927 19:38:37.821826 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:37.825552 (+ 3726us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:37.825594 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:37.826088 (+ 494us) log.cc:842] Serialized 26 byte log entry
0927 19:38:38.070245 (+244157us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:38.156946 (+ 86701us) log.cc:842] Serialized 14 byte log entry
0927 19:38:38.162187 (+ 5241us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.162253 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:38.380804 (+218551us) raft_consensus.cc:1671] finished
0927 19:38:38.380822 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:38.380957 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5142,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":3009,"thread_start_us":299107,"threads_started":2,"wal-append.queue_time_us":501109}
I20260927 19:38:38.170416 4172 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.386359 4172 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.387143 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:38.387936 2685 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:38.389746 4169 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.238184 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.174994 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.155557 3776 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:38.259793 2062 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260927 19:38:38.377295 2037 connection.cc:232] RPC call timeout handler was delayed by -2.0199379250001357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
W20260927 19:38:38.270007 4159 log.cc:924] Time spent T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.319s user 0.000s sys 0.001s
I20260927 19:38:38.395714 2141 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:38.171442 4107 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:38.400669 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1881) took 1159 ms. Trace:
I20260927 19:38:38.397224 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:38.398892 1959 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:38.405164 1962 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.413532 2062 rpcz_store.cc:276] 0927 19:38:37.240909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:37.897652 (+656743us) service_pool.cc:227] Handling call
0927 19:38:37.965004 (+ 67352us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:38.400365 (+435361us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.400603 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:38.250530 3971 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.405637 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.427501 2141 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:38.427861 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.331616 3908 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:38.173110 3988 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:38.400386 2037 connection.cc:232] RPC call timeout handler was delayed by -2.0127379250001241s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
W20260927 19:38:38.435812 2037 connection.cc:232] RPC call timeout handler was delayed by -1.9839379250000775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20260927 19:38:38.326689 4020 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.438362 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:38.439149 2144 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:38.180505 4098 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.255s
I20260927 19:38:38.399673 3776 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.182471 4183 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:38.441480 3988 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.435668 3971 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.449513 1995 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 19:38:38.450042 3988 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.450632 3971 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.450827 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.237514 4043 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.390931 4002 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.051s
I20260927 19:38:38.268646 4102 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.435621 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:38.436339 2037 connection.cc:232] RPC call timeout handler was delayed by -1.8646620289999873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32200000000000006s for a response
I20260927 19:38:38.429457 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:38.465615 2037 connection.cc:232] RPC call timeout handler was delayed by -1.6952944150000349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2683s for a response
I20260927 19:38:38.436662 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.264415 4095 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.180052 3981 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:38.159065 4179 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:38.466145 2037 connection.cc:232] RPC call timeout handler was delayed by -1.6494507449999674s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
I20260927 19:38:38.465852 2141 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:38.467845 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.452481 3776 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:38.433223 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:38.470992 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 113981us
I20260927 19:38:38.471163 1989 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
W20260927 19:38:38.471772 1961 leader_election.cc:334] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.739s (SENT)
I20260927 19:38:38.472097 2686 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:38.472187 1961 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:38.474195 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.475873 2141 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:38.455314 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:38.481606 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.482196 2111 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.482370 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.478542 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.483727 1989 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:38.465580 4048 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:38.461930 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.461616 4020 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.468214 3988 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:38.488823 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:38.491604 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 86508.4us
I20260927 19:38:38.438019 3908 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.493247 1962 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.493693 1993 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:38.494215 3977 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:38.494848 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:38.497720 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 98949.1us
I20260927 19:38:38.500154 1989 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:38.503413 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.504456 2141 raft_consensus.cc:2396] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 8 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260927 19:38:38.486756 4048 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:38.510224 4155 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:38.499552 3833 consensus_queue.cc:1046] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260927 19:38:38.512805 4048 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:38.523964 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1414) took 2579 ms. Trace:
I20260927 19:38:38.524358 2068 rpcz_store.cc:276] 0927 19:38:35.944548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:35.947162 (+ 2614us) service_pool.cc:227] Handling call
0927 19:38:35.947513 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.856925 (+909412us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.523740 (+1666815us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b5400323f54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:38.523910 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1099094272}
I20260927 19:38:38.526515 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1427) took 2507 ms. Trace:
W20260927 19:38:38.526842 4194 log.cc:924] Time spent T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260927 19:38:38.332181 4186 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:38.528020 3978 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.526937 1994 rpcz_store.cc:276] 0927 19:38:36.019247 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.025735 (+ 6488us) service_pool.cc:227] Handling call
0927 19:38:36.039993 (+ 14258us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:36.040220 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:36.040244 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:36.046157 (+ 5913us) log.cc:842] Serialized 26 byte log entry
0927 19:38:36.215797 (+169640us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:36.238983 (+ 23186us) log.cc:842] Serialized 14 byte log entry
0927 19:38:36.239188 (+ 205us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:36.239269 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:36.292158 (+ 52889us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b1800410920. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:38.521633 (+2229475us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b1800410920. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:38.525199 (+ 3566us) raft_consensus.cc:1671] finished
0927 19:38:38.525219 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:38.525339 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9597,"prepare.queue_time_us":22675,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":3154,"raft.queue_time_us":2143,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":13555328,"thread_start_us":140327,"threads_started":1,"wal-append.queue_time_us":189485}
I20260927 19:38:38.528774 3978 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:38.331882 4180 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.513250 4155 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:38.529928 4180 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.514083 4179 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.530485 4155 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:38.532274 4180 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.533751 1992 raft_consensus.cc:1226] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 19:38:38.534358 2688 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 19:38:38.535090 2688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 612) took 2274 ms. Trace:
I20260927 19:38:38.535538 2688 rpcz_store.cc:276] 0927 19:38:36.260296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.261561 (+ 1265us) service_pool.cc:227] Handling call
0927 19:38:38.533635 (+2272074us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:38.533885 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:38.534792 (+ 907us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.535028 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46732032}
I20260927 19:38:38.540555 2142 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:38.264743 4188 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.516454 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:38.542229 4197 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20260927 19:38:38.490541 3988 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:38.543134 2680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 615) took 2018 ms. Trace:
I20260927 19:38:38.543346 3988 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:38.528055 4186 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.243057 4187 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.366s
I20260927 19:38:38.546406 2065 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1853) took 2204 ms. Trace:
I20260927 19:38:38.546912 2065 rpcz_store.cc:276] 0927 19:38:36.342245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.929755 (+587510us) service_pool.cc:227] Handling call
0927 19:38:36.930121 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:37.568420 (+638299us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:37.568461 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:37.568529 (+ 68us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:37.568574 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.546110 (+977536us) spinlock_profiling.cc:243] Waited 638 ms on lock 0x7b54002a4c54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:38.546172 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:38.546337 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1467180416}
I20260927 19:38:38.549870 3830 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:38.551642 4171 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:38.552052 4171 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:38.516100 3908 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.554281 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.555266 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 1101) took 1001 ms. Trace:
I20260927 19:38:38.555619 2136 rpcz_store.cc:276] 0927 19:38:37.554423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:37.558446 (+ 4023us) service_pool.cc:227] Handling call
0927 19:38:38.554005 (+995559us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:38.555201 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18800896}
I20260927 19:38:38.552450 4171 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:38.562798 4164 raft_consensus.cc:979] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:38.512673 3977 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:37.725908 2069 rpcz_store.cc:276] 0927 19:38:36.443089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.933289 (+490200us) service_pool.cc:227] Handling call
0927 19:38:36.938165 (+ 4876us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.938388 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 19:38:37.166907 (+228519us) log.cc:842] Serialized 14 byte log entry
0927 19:38:37.724269 (+557362us) log.cc:842] Serialized 14 byte log entry
0927 19:38:37.724455 (+ 186us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:37.724497 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:37.724543 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:37.724604 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:37.724713 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":48528,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":4130,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":14897,"spinlock_wait_cycles":15990016,"thread_start_us":709865,"threads_started":2}
I20260927 19:38:38.569867 4202 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:38.572031 1989 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:38.571548 4188 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:38.573065 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:38.573657 4196 log.cc:924] Time spent T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.232s user 0.003s sys 0.000s
I20260927 19:38:38.574237 4095 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:38.575439 4098 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:38.575968 4098 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:38.576397 4098 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:38.578402 4043 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:38.582494 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1095) took 1053 ms. Trace:
I20260927 19:38:38.583045 2143 rpcz_store.cc:276] 0927 19:38:37.528634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:37.528778 (+ 144us) service_pool.cc:227] Handling call
0927 19:38:37.549146 (+ 20368us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:37.568224 (+ 19078us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.582167 (+1013943us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b5400141754. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:38.582415 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42082048}
I20260927 19:38:38.495958 4020 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:38.586668 4190 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.482s user 0.002s sys 0.000s
I20260927 19:38:38.588456 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:38.589984 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 72182.3us
W20260927 19:38:38.591544 2654 leader_election.cc:334] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): Timed out: RequestConsensusVote RPC to 127.0.204.131:45045 timed out after 1.993s (SENT)
I20260927 19:38:38.591859 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.592104 2654 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:38.592878 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1850) took 2324 ms. Trace:
I20260927 19:38:38.593274 2064 rpcz_store.cc:276] 0927 19:38:36.268923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.286990 (+ 18067us) service_pool.cc:227] Handling call
0927 19:38:36.292121 (+ 5131us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:36.310884 (+ 18763us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.592669 (+2281785us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b5400322654. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:38.592826 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41478528}
I20260927 19:38:38.593736 3834 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:38:38.591981 2070 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1864) took 2104 ms (client timeout 1739 ms). Trace:
W20260927 19:38:38.594973 2070 rpcz_store.cc:269] 0927 19:38:36.487145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.770312 (+283167us) service_pool.cc:227] Handling call
0927 19:38:38.591796 (+1821484us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b5400323a54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:38.591916 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111256960}
I20260927 19:38:38.595585 3833 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:38.596083 3833 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:38.605767 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 61806.4us
I20260927 19:38:38.610466 2685 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:38.543582 2680 rpcz_store.cc:276] 0927 19:38:36.524538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.852679 (+328141us) service_pool.cc:227] Handling call
0927 19:38:36.853161 (+ 482us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:37.558905 (+705744us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:37.562022 (+ 3117us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:37.577122 (+ 15100us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:37.581552 (+ 4430us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.542897 (+961345us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54006ed054. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:38.542949 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:38.543071 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1593269504}
W20260927 19:38:38.620282 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 442464us
I20260927 19:38:38.639674 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:38.641182 2110 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:38.641952 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.643179 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.591565 4132 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.657444 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:38.647836 3908 raft_consensus.cc:1067] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:38.650128 2681 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:38.661027 4179 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:38.668702 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:38.667786 2653 leader_election.cc:334] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.632s (SENT)
I20260927 19:38:38.673288 2653 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:38.658741 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:38.691310 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 43765.7us
W20260927 19:38:38.683205 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 189908us
I20260927 19:38:38.675230 4179 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:38.717139 1962 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.661545 3776 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:38.676618 4020 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.729054 4207 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:38.746776 4207 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:38.747273 4207 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:38.747762 3776 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.718510 4106 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.748592 4106 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.733070 4208 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:38.751498 4172 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:38.752525 4183 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:38.753511 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:38.753775 4063 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:38.755309 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.755417 4063 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:38.755931 2143 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:38.715713 4181 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:38.762251 2144 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:38.741752 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:38.721009 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.692894 4186 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.718623 4188 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:38.690805 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.727293 4179 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:38.745596 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:38.725317 4161 consensus_meta.cc:287] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.002s
W20260927 19:38:38.741372 4176 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.672s user 0.003s sys 0.000s
I20260927 19:38:38.768851 2145 raft_consensus.cc:2396] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:38.650341 3833 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:38.779886 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 278712us
I20260927 19:38:38.729099 4026 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:38.794605 4164 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:38.804593 3977 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:38.816999 3833 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:38.805846 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 303037us
I20260927 19:38:38.806804 1962 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:38.804500 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 326283us
W20260927 19:38:38.812646 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 329784us
W20260927 19:38:38.820339 4174 consensus_meta.cc:287] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.923s user 0.002s sys 0.008s
I20260927 19:38:38.813746 2680 raft_consensus.cc:2379] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 2: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:38.817570 4026 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:38.834487 2108 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.837503 1959 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:38.838250 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 115313us
I20260927 19:38:38.821844 4132 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:38.840158 1959 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:38.855713 2110 leader_election.cc:334] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.952s (SENT)
I20260927 19:38:38.856268 2110 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:38.824973 4106 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:38.886850 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46832us
I20260927 19:38:38.809841 4161 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:38.896816 1961 leader_election.cc:334] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.785s (SENT)
I20260927 19:38:38.890220 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.839318 4026 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:38.905974 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 50296.9us
I20260927 19:38:38.907624 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.831817 4107 raft_consensus.cc:686] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:38.909806 2145 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
W20260927 19:38:38.887590 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 46126.5us
I20260927 19:38:38.872356 3924 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:38.873018 3776 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:38.872026 4191 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:38.942081 2111 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.900553 1961 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:38.831238 4048 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:38.948840 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 76872.1us
I20260927 19:38:38.835717 4155 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.959761 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:38.839787 4174 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:38.946769 4095 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:38.945860 4180 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:39.032600 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 120236us
W20260927 19:38:39.033556 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 75230.2us
I20260927 19:38:38.943778 4106 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:39.046965 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:39.047581 2110 leader_election.cc:334] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.069s (SENT)
I20260927 19:38:38.840669 4203 consensus_queue.cc:1046] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.111s
I20260927 19:38:38.964233 4186 raft_consensus.cc:2781] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:39.053225 2685 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:38.953140 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:39.064898 2111 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:38.964600 4004 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:39.070356 2111 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:39.050798 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 98713us
I20260927 19:38:38.951334 3776 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:38.887046 4212 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
W20260927 19:38:38.960664 4206 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.236s user 0.001s sys 0.000s
W20260927 19:38:39.051932 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 155176us
W20260927 19:38:39.082518 1961 leader_election.cc:334] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.813s (SENT)
W20260927 19:38:39.078404 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 238316us
I20260927 19:38:39.030638 4063 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:39.050588 4165 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:39.062201 3990 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:39.089167 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.090353 1959 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:38.943373 4210 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.127s
I20260927 19:38:39.079582 3776 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:39.079850 4102 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:39.061852 3908 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260927 19:38:39.073825 4204 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:39.051501 4095 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.053778 4183 consensus_queue.cc:1046] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260927 19:38:39.056425 4106 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.104200 3888 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:39.111510 4102 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:39.096874 4004 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260927 19:38:39.074466 2683 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
W20260927 19:38:39.140072 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 57601.8us
W20260927 19:38:39.135471 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 64291.6us
W20260927 19:38:39.122063 4209 log.cc:924] Time spent T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.309s user 0.001s sys 0.000s
W20260927 19:38:39.144061 1961 leader_election.cc:334] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.818s (SENT)
I20260927 19:38:39.145892 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:39.146592 1996 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 5
I20260927 19:38:39.147382 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:39.117164 2679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 619) took 1415 ms. Trace:
I20260927 19:38:39.127980 4204 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:39.126955 4191 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:39.162273 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:39.164527 4109 consensus_meta.cc:287] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 1.289s user 0.006s sys 0.006s
W20260927 19:38:39.165282 4184 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 1.121s user 0.003s sys 0.000s
I20260927 19:38:39.167035 4048 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:39.167830 4095 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.085873 4107 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.153000 2679 rpcz_store.cc:276] 0927 19:38:37.696961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:37.702245 (+ 5284us) service_pool.cc:227] Handling call
0927 19:38:37.745220 (+ 42975us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:37.768455 (+ 23235us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:37.768496 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:37.769074 (+ 578us) log.cc:842] Serialized 26 byte log entry
0927 19:38:38.023578 (+254504us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:38.028596 (+ 5018us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:39.070077 (+1041481us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b5400430654. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:39.074041 (+ 3964us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:39.076235 (+ 2194us) raft_consensus.cc:1671] finished
0927 19:38:39.079904 (+ 3669us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:39.111982 (+ 32078us) inbound_call.cc:173] Queueing success response
0927 19:38:39.113596 (+ 1614us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b180038a520. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
Metrics: {"raft.queue_time_us":8440,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":11576,"spinlock_wait_cycles":80644224,"thread_start_us":174351,"threads_started":1,"wal-append.queue_time_us":587217}
W20260927 19:38:39.172374 1961 leader_election.cc:334] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.850s (SENT)
I20260927 19:38:39.172969 1961 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.176659 1959 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.176983 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:39.094785 4063 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.144929 4102 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:39.188405 2108 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:39.190650 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:39.181200 3908 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:39.198338 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.199072 2142 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:39.177762 2685 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:39.220634 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.172222 4204 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:39.220995 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 96264.7us
I20260927 19:38:39.221563 2143 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:39.224390 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:39.214011 1959 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:39.219789 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.205909 1996 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:39.224777 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:39.219157 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46834us
W20260927 19:38:39.226550 1961 leader_election.cc:334] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.852s (SENT)
I20260927 19:38:39.226996 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.219451 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.223894 2654 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:39.190294 4098 consensus_meta.cc:287] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.613s user 0.011s sys 0.000s
I20260927 19:38:39.229203 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.221625 4155 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:39.230508 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:39.201460 4161 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.218658 4188 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:39.213065 4214 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:39.212952 4191 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:39.264025 1961 leader_election.cc:334] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.856s (SENT)
I20260927 19:38:39.264724 1961 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.236382 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.242197 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:39.220727 3977 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:39.236064 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.219852 4230 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:39.221210 4217 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260927 19:38:39.247042 4222 log.cc:924] Time spent T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260927 19:38:39.276374 4188 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.304301 4188 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.305254 4191 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.249209 4153 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:39.250702 2066 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:39.251247 4226 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260927 19:38:39.221297 3908 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.318917 4132 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:39.253468 4214 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:39.254029 4171 consensus_meta.cc:287] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.696s user 0.012s sys 0.003s
I20260927 19:38:39.235270 4109 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:39.258553 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1098) took 1303 ms. Trace:
I20260927 19:38:39.274608 2143 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:39.231875 4174 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.233492 4004 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:39.233913 4043 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:39.275679 4155 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 19:38:39.298545 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 55057.6us
I20260927 19:38:39.284265 4230 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:39.299335 4216 log.cc:924] Time spent T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.248s user 0.001s sys 0.001s
W20260927 19:38:39.299499 3988 consensus_meta.cc:287] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.756s user 0.011s sys 0.000s
I20260927 19:38:39.300817 2142 raft_consensus.cc:1226] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:39.365713 2143 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:39.309424 2145 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:39.319195 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 75598.4us
I20260927 19:38:39.236189 4098 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.331844 4204 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.239965 3924 raft_consensus.cc:686] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:39.245236 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:39.344915 3908 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:39.347692 4171 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.356532 4043 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:39.300819 4063 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:39.348341 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 134334us
I20260927 19:38:39.350288 1988 rpcz_store.cc:276] 0927 19:38:37.955281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.154967 (+199686us) service_pool.cc:227] Handling call
0927 19:38:38.164654 (+ 9687us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:38.164991 (+ 337us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:38.165018 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:38.170475 (+ 5457us) log.cc:842] Serialized 26 byte log entry
0927 19:38:38.474647 (+304172us) raft_consensus.cc:1631] Marking committed up to 1
0927 19:38:38.474682 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.474743 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:39.258231 (+783488us) raft_consensus.cc:1671] finished
0927 19:38:39.258260 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:39.258465 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":214,"raft.queue_time_us":503,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":4728,"thread_start_us":124663,"threads_started":1,"wal-append.queue_time_us":458394}
I20260927 19:38:39.317961 3917 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:39.317744 4153 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:39.333120 4132 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:39.363974 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 120511us
I20260927 19:38:39.365417 4230 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:39.370916 4106 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.349951 4109 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:39.375550 4004 consensus_queue.cc:1046] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:39.345813 4214 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:39.379971 4207 consensus_meta.cc:287] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.630s user 0.013s sys 0.000s
I20260927 19:38:39.385603 2684 raft_consensus.cc:1203] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Deduplicated request from leader. Original: 3.5->[4.6-6.7] Dedup: 4.6->[6.7-6.7]
I20260927 19:38:39.366415 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.392966 2654 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:39.472519 2684 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
W20260927 19:38:39.472802 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 79915.5us
W20260927 19:38:39.472977 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 106843us
I20260927 19:38:39.366255 4155 raft_consensus.cc:686] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:39.400216 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:39.401899 3924 consensus_queue.cc:237] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:39.404857 4179 consensus_meta.cc:287] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.003s
I20260927 19:38:39.461949 4109 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:39.392014 4153 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:39.492942 4098 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:39.420117 4204 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:39.455929 2687 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 391) took 1053 ms. Trace:
I20260927 19:38:39.495107 2687 rpcz_store.cc:276] 0927 19:38:38.402301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.426828 (+ 24527us) service_pool.cc:227] Handling call
0927 19:38:38.441730 (+ 14902us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:38.442199 (+ 469us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:38.442238 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:38.461023 (+ 18785us) log.cc:842] Serialized 26 byte log entry
0927 19:38:38.825492 (+364469us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:38.825538 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:38.831653 (+ 6115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:39.455552 (+623899us) raft_consensus.cc:1671] finished
0927 19:38:39.455589 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:39.455835 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":611,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":11288,"thread_start_us":338533,"threads_started":1,"wal-append.queue_time_us":436816}
I20260927 19:38:39.386672 4043 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:39.461450 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 66534us
W20260927 19:38:39.462638 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 105322us
I20260927 19:38:39.497948 4155 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:39.499254 2653 leader_election.cc:334] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.607s (SENT)
W20260927 19:38:39.467566 4215 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.507s user 0.003s sys 0.000s
I20260927 19:38:39.502482 4171 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:39.503607 2653 leader_election.cc:334] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.728s (SENT)
I20260927 19:38:39.504042 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:39.505034 4240 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 19:38:39.394788 4026 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.476643 4241 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:39.509418 4179 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.510185 4241 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:39.510680 4241 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:39.512199 4043 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.514954 4026 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.515762 4153 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.518253 4102 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.473280 2684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 389) took 1108 ms. Trace:
I20260927 19:38:39.519982 2684 rpcz_store.cc:276] 0927 19:38:38.364880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.367100 (+ 2220us) service_pool.cc:227] Handling call
0927 19:38:38.385781 (+ 18681us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:39.362446 (+976665us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b5400474120. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138f2730 00007f7913964300 00007f7913965371 00007f7913961edf 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc
0927 19:38:39.472978 (+110532us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:39.473220 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":186319744}
I20260927 19:38:39.473759 4174 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:39.474334 2654 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.522406 4240 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:39.522595 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48333us
I20260927 19:38:39.523543 2654 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.524554 4240 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:38:39.366981 1961 leader_election.cc:334] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.939s (SENT)
I20260927 19:38:39.525638 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:39.526918 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 159946us
W20260927 19:38:39.527746 1961 leader_election.cc:334] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.000s (SENT)
I20260927 19:38:39.528242 1961 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.370476 3988 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.475924 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:38:39.529776 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 53915.8us
I20260927 19:38:39.530745 3988 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.530849 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:39.530962 2140 raft_consensus.cc:1226] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
W20260927 19:38:39.405323 2066 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48556 (request call id 1873) took 2372 ms (client timeout 2220 ms). Trace:
W20260927 19:38:39.532058 2066 rpcz_store.cc:269] 0927 19:38:37.033632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:37.651478 (+617846us) service_pool.cc:227] Handling call
0927 19:38:39.405004 (+1753526us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b5400323554. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:39.405199 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3478137600}
W20260927 19:38:39.532673 1961 leader_election.cc:334] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.043s (SENT)
I20260927 19:38:39.519157 4153 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.524945 2654 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:39.533089 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:39.470860 3776 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.549500 4232 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:39.550895 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:39.475383 4207 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.549012 2137 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:39.558405 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.561400 2143 raft_consensus.cc:1226] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:39.565284 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:39.565951 2144 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:39.568356 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:39.572816 2143 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:39.546175 3917 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:39.556252 4165 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:39.518296 4043 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:39.580331 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:39.573788 4153 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:39.556530 4240 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:39.579629 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:39.600593 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 67936.2us
I20260927 19:38:39.605852 2137 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:39.629563 2143 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:39.634044 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 103282us
I20260927 19:38:39.631558 2144 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:39.561188 4174 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20260927 19:38:39.605626 4240 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:38:39.637410 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 67462.8us
I20260927 19:38:39.639559 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:39.645325 1962 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:39.598932 4165 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:39.605235 2138 raft_consensus.cc:1226] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:39.561028 4173 consensus_queue.cc:1046] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260927 19:38:39.646169 4249 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:39.659459 3924 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:39.696264 4249 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:39.696672 3924 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:39.697767 3924 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.636662 4207 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:39.690021 2653 leader_election.cc:334] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.896s (SENT)
I20260927 19:38:39.696712 4249 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:39.653921 2143 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:39.711812 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:39.712744 2143 raft_consensus.cc:595] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:39.713173 2143 raft_consensus.cc:603] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:39.659970 4165 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:39.653155 4240 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:39.731351 2654 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:39.731904 2687 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:39.598353 4063 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:38:39.767673 2653 leader_election.cc:334] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.993s (SENT)
I20260927 19:38:39.768366 2653 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:39.779110 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 144350us
I20260927 19:38:39.783151 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39d534ed586444279970bb03de60c919"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:39.784060 2143 raft_consensus.cc:595] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:39.784526 2143 raft_consensus.cc:603] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:39.746363 4102 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:39.755932 4191 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:39.793707 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 189845us
I20260927 19:38:39.790764 4252 consensus_queue.cc:1046] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
W20260927 19:38:39.801664 4246 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260927 19:38:39.802191 4252 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:39.802996 4252 raft_consensus.cc:3032] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:39.803395 4252 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:39.804122 4252 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.806162 4250 raft_consensus.cc:3032] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:39.805944 4191 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.806516 4250 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:39.807350 4250 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.808472 4191 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:39.813227 2653 leader_election.cc:334] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.035s (SENT)
I20260927 19:38:39.814477 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.784756 4173 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:39.817063 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.820763 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.817284 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.818071 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.818693 2143 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:39.819463 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.824795 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.821261 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:39.822366 2653 leader_election.cc:334] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.042s (SENT)
I20260927 19:38:39.826040 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:39.825560 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 77877.3us
I20260927 19:38:39.829756 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.833879 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:39.834383 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:39.834558 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.808840 4250 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:39.835747 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.815830 4252 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:39.837507 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.838687 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:39.839812 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:39.841985 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.842583 1996 raft_consensus.cc:1226] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:39.815925 3776 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:39.843643 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:39.849182 2143 raft_consensus.cc:595] T 3d15dfddc2cb4edfaef7625c7eba492c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:38:39.844208 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 63671us
W20260927 19:38:39.874349 4186 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.012s sys 0.009s
W20260927 19:38:39.924803 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 102557us
W20260927 19:38:39.923321 4186 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.837s user 0.020s sys 0.011s
I20260927 19:38:39.831405 4218 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260927 19:38:39.843813 4153 consensus_queue.cc:1046] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260927 19:38:39.942842 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:39.908108 4248 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260927 19:38:39.894881 3888 raft_consensus.cc:3032] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:39.891853 4257 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:39.956182 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:39.890003 4155 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:39.959192 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 107449us
I20260927 19:38:39.960194 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:39.962531 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 104142us
I20260927 19:38:39.946748 4186 raft_consensus.cc:686] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:39.961591 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.966076 1992 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:39.943352 4173 raft_consensus.cc:686] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:39.916527 4249 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.957687 4181 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:39.953603 4251 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:39.954926 3888 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:39.975903 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:39.977504 1992 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:39.992401 4186 consensus_queue.cc:237] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.994179 3888 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:39.995640 3888 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:39.997726 4230 consensus_meta.cc:287] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.545s user 0.005s sys 0.005s
I20260927 19:38:39.998142 4230 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:39.999624 4132 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.000097 4230 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.000226 4132 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.000764 4264 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:39.960786 4257 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.009332 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.009976 1992 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:40.013744 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 53624.2us
I20260927 19:38:40.014465 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:39.974711 4260 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260927 19:38:40.023964 2037 leader_election.cc:398] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:40.027284 2143 raft_consensus.cc:595] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:40.027889 2143 raft_consensus.cc:603] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:40.028687 3981 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:40.028940 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2068) took 2024 ms. Trace:
I20260927 19:38:40.029218 3981 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.029259 2139 rpcz_store.cc:276] 0927 19:38:38.005047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.488435 (+483388us) service_pool.cc:227] Handling call
0927 19:38:38.488865 (+ 430us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:38.649275 (+160410us) raft_consensus.cc:1520] Early marking committed up to index 7
0927 19:38:38.823773 (+174498us) log.cc:842] Serialized 14 byte log entry
0927 19:38:39.071318 (+247545us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:39.079485 (+ 8167us) log.cc:842] Serialized 26 byte log entry
0927 19:38:39.079648 (+ 163us) raft_consensus.cc:1631] Marking committed up to 7
0927 19:38:39.079687 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:40.008623 (+928936us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b18004125a0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:40.028656 (+ 20033us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400051254. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:40.028710 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:40.028754 (+ 44us) raft_consensus.cc:1671] finished
0927 19:38:40.028767 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:40.028889 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55156,"prepare.queue_time_us":424746,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":46737,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":14534,"spinlock_wait_cycles":272722048,"thread_start_us":312731,"threads_started":2,"wal-append.queue_time_us":333209}
I20260927 19:38:40.031453 3981 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.027429 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2085) took 1289 ms. Trace:
I20260927 19:38:40.033855 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56618 (request call id 300) took 1280 ms. Trace:
I20260927 19:38:40.034225 1989 rpcz_store.cc:276] 0927 19:38:38.753028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.753224 (+ 196us) service_pool.cc:227] Handling call
0927 19:38:40.033683 (+1280459us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540036f454. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:40.033804 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1397227776}
I20260927 19:38:40.018009 4132 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:40.039049 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.033245 2141 rpcz_store.cc:276] 0927 19:38:38.738318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.738470 (+ 152us) service_pool.cc:227] Handling call
0927 19:38:38.738756 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:38.738946 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 19:38:38.944738 (+205792us) log.cc:842] Serialized 14 byte log entry
0927 19:38:39.553011 (+608273us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b1800412480. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:40.026894 (+473883us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b1800412480. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f79135f182c 00007f7910a1c9dc 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a37155 00007f791395a3d0 00007f791396c576 00007f791397d0d7
0927 19:38:40.027109 (+ 215us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:40.027145 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:40.027173 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:40.027253 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:40.027378 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8496,"prepare.queue_time_us":394704,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":48388,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":8038,"spinlock_wait_cycles":36424192,"thread_start_us":381912,"threads_started":3,"wal-append.queue_time_us":222592}
I20260927 19:38:40.058862 4257 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.055639 4132 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:40.071899 4132 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:40.080317 4179 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:40.083859 4109 consensus_meta.cc:287] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.537s user 0.010s sys 0.000s
I20260927 19:38:40.086213 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:40.090067 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.091027 2679 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:40.094007 4254 log.cc:924] Time spent T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260927 19:38:40.094761 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 78672.3us
I20260927 19:38:40.063479 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.062752 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:40.064872 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.064591 4240 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.037032 1962 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.064201 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:40.094165 4241 consensus_meta.cc:287] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.583s user 0.009s sys 0.002s
I20260927 19:38:40.097663 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.100584 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.096491 1996 raft_consensus.cc:2354] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7.
W20260927 19:38:40.104195 1961 leader_election.cc:334] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.967s (SENT)
I20260927 19:38:40.106585 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.108696 2111 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.084612 4109 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.111176 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.038370 4173 consensus_queue.cc:237] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.115911 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.129243 2111 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.130316 1992 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:40.135470 2111 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.110526 4241 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.140161 3776 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.140455 4241 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.109163 4205 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:40.141886 4180 raft_consensus.cc:3032] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:40.142316 4180 raft_consensus.cc:729] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:40.143244 4180 consensus_queue.cc:260] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.144372 4180 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:40.145890 2138 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2093) took 1257 ms. Trace:
I20260927 19:38:40.146332 2138 rpcz_store.cc:276] 0927 19:38:38.888871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.888978 (+ 107us) service_pool.cc:227] Handling call
0927 19:38:39.397303 (+508325us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02bce 000000000034aea9 00007f791882ea30 00007f79188f47b4 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:39.397486 (+ 183us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:39.662680 (+265194us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:40.145604 (+482924us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400075dd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:40.145822 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":444263936}
I20260927 19:38:40.148543 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2095) took 1249 ms. Trace:
I20260927 19:38:40.149075 2136 rpcz_store.cc:276] 0927 19:38:38.899279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:38.906854 (+ 7575us) service_pool.cc:227] Handling call
0927 19:38:40.147686 (+1240832us) spinlock_profiling.cc:243] Waited 5.07 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:40.147932 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:40.148139 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:40.148174 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:40.148231 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:40.148269 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:40.148336 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:40.148483 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11661696}
I20260927 19:38:40.150360 4063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33206 (request call id 37) took 3878 ms. Trace:
I20260927 19:38:40.140884 4240 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:40.160305 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 313434us
I20260927 19:38:40.141167 4164 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260927 19:38:40.163800 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 58186.8us
W20260927 19:38:40.164661 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 60506us
W20260927 19:38:40.167743 1961 leader_election.cc:334] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.238s (SENT)
I20260927 19:38:40.132352 4189 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.169625 1992 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:40.170943 1961 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:40.166443 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 129489us
I20260927 19:38:40.134158 4249 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.154599 2111 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.174268 2111 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.168675 4217 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:40.187626 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 71780.7us
I20260927 19:38:40.188400 1994 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:40.190968 4109 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.189019 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.192718 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.193153 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:40.192957 4132 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.193719 2140 raft_consensus.cc:3037] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:40.195962 2138 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:40.150771 4063 rpcz_store.cc:276] 0927 19:38:36.272479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:36.282053 (+ 9574us) service_pool.cc:227] Handling call
0927 19:38:37.055267 (+773214us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b58007331b0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138ddf63 00007f7913941aba 00007f791395da7a 00007f791395eada 00007f791395dc8b 00007f7913969084 00007f7913968743 00007f7913960c1a 00007f79188a46d9 00007f7911f8e184
0927 19:38:37.564413 (+509146us) log.cc:842] Serialized 501 byte log entry
0927 19:38:38.518163 (+953750us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b1800080700. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f791399ef11 00007f79138f3f91 00007f79138dc545 00007f79138e1771 00007f79138e16f9 00007f79138e1689 00007f79138e1651 00007f79138e081d 00007f7919c9e204 00007f7919c9e039
0927 19:38:39.156376 (+638213us) spinlock_profiling.cc:243] Waited 41.7 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:40.150251 (+993875us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":1334,"mutex_wait_us":79012,"raft.queue_time_us":91919,"raft.run_cpu_time_us":7381,"raft.run_wall_time_us":704209,"spinlock_wait_cycles":132142336,"thread_start_us":162881,"threads_started":1,"wal-append.queue_time_us":342791}
I20260927 19:38:40.197896 2136 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8. Ops: []
I20260927 19:38:40.199478 4269 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:40.200021 4269 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:40.200498 4269 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.178128 4164 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:40.201813 4164 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:40.202234 4164 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.170270 4167 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:40.205830 4257 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:40.206713 4257 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:40.208570 4257 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:40.209425 4048 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.189291 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:40.211184 2143 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.187723 4249 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:40.213745 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46018.9us
I20260927 19:38:40.169866 4266 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:38:40.216120 4184 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260927 19:38:40.224458 2140 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:40.227792 2143 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:40.161231 4165 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.176011 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.232859 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.219700 3776 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:40.227630 4167 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.237545 1935 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 11)
I20260927 19:38:40.237815 4167 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.238832 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.239184 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.239673 2684 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:40.241472 4179 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:40.242707 3776 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:40.243312 3776 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:40.243685 4165 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.219830 4257 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260927 19:38:40.246992 2111 leader_election.cc:398] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7.
I20260927 19:38:40.247309 2111 leader_election.cc:401] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260927 19:38:40.249171 4189 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7.
I20260927 19:38:40.252375 2111 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.262173 3978 consensus_queue.cc:1046] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:38:40.263090 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:40.225795 4132 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.269433 4189 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:40.270831 4189 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:40.272126 4269 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.271982 4189 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.275743 4218 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:40.275828 3888 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:38:40.277439 4208 raft_consensus.cc:3032] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:40.277827 4208 raft_consensus.cc:729] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.278622 4208 consensus_queue.cc:260] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.279963 4208 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.286811 4269 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:40.286994 1961 leader_election.cc:334] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.220s (SENT)
W20260927 19:38:40.292757 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 97783.3us
I20260927 19:38:40.301745 4234 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:38:40.361573 4234 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.362788 4234 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.364642 4234 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:40.305540 2654 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:40.365834 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 60368.5us
I20260927 19:38:40.307188 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:40.366915 2654 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.340942 2687 raft_consensus.cc:2396] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260927 19:38:40.369179 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.371794 4173 raft_consensus.cc:2781] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:40.372550 4173 raft_consensus.cc:686] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:40.289906 4164 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.342561 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.374387 2688 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.373598 4173 consensus_queue.cc:237] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.375604 4164 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:40.377588 4261 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:40.343276 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46930 (request call id 308) took 1129 ms. Trace:
I20260927 19:38:40.378780 2145 rpcz_store.cc:276] 0927 19:38:39.213685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:39.219549 (+ 5864us) service_pool.cc:227] Handling call
0927 19:38:40.343106 (+1123557us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b5400225954. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:40.343213 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191788928}
I20260927 19:38:40.379917 2108 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.343683 2142 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:40.380777 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.381850 3981 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.383671 4252 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:40.387236 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:40.353277 3776 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.305683 4150 consensus_queue.cc:1046] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:40.378317 4261 raft_consensus.cc:686] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:40.389231 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.389858 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.355621 2679 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:40.390708 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:40.392102 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.405969 2071 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:40.395493 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.399001 2069 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:40.401132 2108 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.408895 2108 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.390149 2065 raft_consensus.cc:1226] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 3. Current term is 5. Ops: []
I20260927 19:38:40.412532 2070 raft_consensus.cc:1226] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:40.412886 4150 raft_consensus.cc:3032] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:40.413973 2071 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:40.413791 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.414458 2064 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:40.415185 2066 raft_consensus.cc:1226] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260927 19:38:40.419111 2067 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:40.418634 2688 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:40.419960 2136 raft_consensus.cc:595] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:40.423789 2136 raft_consensus.cc:603] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:40.415705 4187 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:38:40.405618 4048 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260927 19:38:40.425630 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.426326 2062 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:40.428364 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.429167 1959 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:40.429283 2068 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:40.428668 1892 catalog_manager.cc:5495] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:40.430166 1959 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:40.430433 4048 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:40.431779 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:38:40.437062 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:40.438138 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 19:38:40.419533 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 132559us
W20260927 19:38:40.441856 1961 leader_election.cc:334] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.238s (SENT)
I20260927 19:38:40.446889 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.447688 1992 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.450206 4261 consensus_queue.cc:237] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.405277 4043 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:40.453722 4043 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:40.454118 4043 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:40.458776 4048 raft_consensus.cc:686] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.460008 4048 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.420034 4102 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:40.463699 3776 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:40.420439 3990 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:40.417713 4150 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:40.466262 4150 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.467679 4150 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:40.435367 4252 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:40.469439 4252 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.472786 4102 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:40.473456 4102 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.464385 3971 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:40.450364 1892 catalog_manager.cc:5495] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 4, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:40.490630 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.491353 2067 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:40.499588 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.500571 2071 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:40.508647 2143 raft_consensus.cc:1226] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:40.507489 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:40.504390 1891 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 255) took 14766 ms (client timeout 15000 ms). Trace:
I20260927 19:38:40.509093 1992 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:40.522617 2069 raft_consensus.cc:1261] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:40.522316 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.524128 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.527742 2066 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:40.524196 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.525074 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.528919 2068 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:40.525204 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.529856 2063 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:40.527527 2071 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:40.536302 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.537091 4102 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.537158 2065 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:40.537907 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.538580 2063 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:40.539784 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.540387 2070 raft_consensus.cc:2362] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5. Re-sending same reply.
I20260927 19:38:40.541574 4252 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.497323 3971 raft_consensus.cc:686] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.544624 3971 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:40.538589 4280 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260927 19:38:40.544939 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:40.553972 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:40.555190 2143 raft_consensus.cc:2396] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260927 19:38:40.559177 2064 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:40.559659 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:40.560518 2688 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
W20260927 19:38:40.513729 1891 rpcz_store.cc:269] 0927 19:38:25.737616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:25.737872 (+ 256us) service_pool.cc:227] Handling call
0927 19:38:40.504297 (+14766425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:38.486989 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:38.566047 (+ 79058us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:38.566112 (+ 65us) write_op.cc:435] Acquired schema lock
0927 19:38:38.566133 (+ 21us) tablet.cc:660] Decoding operations
0927 19:38:38.566894 (+ 761us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:38.567010 (+ 116us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:38.567047 (+ 37us) tablet.cc:683] Acquiring locks for 3 operations
0927 19:38:38.567497 (+ 450us) tablet.cc:699] Row locks acquired
0927 19:38:38.567519 (+ 22us) write_op.cc:260] PREPARE: finished
0927 19:38:38.567699 (+ 180us) write_op.cc:270] Start()
0927 19:38:38.567847 (+ 148us) write_op.cc:276] Timestamp: P: 1790537918567627 usec, L: 0
0927 19:38:38.567889 (+ 42us) op_driver.cc:331] REPLICATION: starting
0927 19:38:38.568659 (+ 770us) log.cc:842] Serialized 1317 byte log entry
0927 19:38:39.391088 (+822429us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:39.703849 (+312761us) op_driver.cc:444] REPLICATION: finished
0927 19:38:39.955252 (+251403us) write_op.cc:301] APPLY: starting
0927 19:38:39.955381 (+ 129us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:39.976097 (+ 20716us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:39.978083 (+ 1986us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:39.989870 (+ 11787us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:39.990146 (+ 276us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0927 19:38:39.990217 (+ 71us) write_op.cc:312] APPLY: finished
0927 19:38:39.990576 (+ 359us) log.cc:842] Serialized 35 byte log entry
0927 19:38:39.990760 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:39.990857 (+ 97us) write_op.cc:454] Released schema lock
0927 19:38:39.990898 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:39.991159 (+ 261us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:39.991188 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":87,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":240445,"apply.run_cpu_time_us":3069,"apply.run_wall_time_us":42010,"num_ops":3,"prepare.queue_time_us":188633,"prepare.run_cpu_time_us":25783,"prepare.run_wall_time_us":353239,"raft.queue_time_us":201330,"raft.run_cpu_time_us":20714,"raft.run_wall_time_us":81698,"replication_time_us":1135853,"spinlock_wait_cycles":6590720,"thread_start_us":612883,"threads_started":4,"wal-append.queue_time_us":325203}]]}
I20260927 19:38:40.576083 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.577010 4260 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:40.577323 2108 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.577701 4260 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.578780 3776 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:40.579346 3776 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:40.579784 3776 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.580084 4260 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.582651 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.583282 2070 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:40.584944 2143 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:40.592147 2067 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:40.595619 4173 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:40.623720 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.624485 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.624516 2064 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:40.604609 4172 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260927 19:38:40.636829 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.637782 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.638804 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.641261 2070 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:40.642360 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:40.642527 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:40.642827 2065 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:40.641762 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.643003 2063 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:40.644172 2069 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:40.645081 2037 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.649500 4257 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.650205 4043 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.654048 4043 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:40.655195 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.661535 4260 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:40.668597 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.669250 1996 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:40.671327 1992 raft_consensus.cc:1261] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:40.671774 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:40.676462 3776 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.679378 2688 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 19:38:40.679421 1961 leader_election.cc:334] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.485s (SENT)
I20260927 19:38:40.681166 2140 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:40.681320 4234 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:40.684019 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.684669 3917 consensus_queue.cc:1046] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:40.685667 3917 consensus_queue.cc:1225] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer 69f09866d0a1477585f05b2f82ff4c67 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 19:38:40.685089 4257 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:40.689827 2140 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:40.684203 2064 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:40.697300 2111 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.699421 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.700949 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.701813 4257 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.703531 4180 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:40.704082 4208 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:40.704094 4180 raft_consensus.cc:686] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:40.709134 4218 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:40.709837 4218 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.709405 4180 consensus_queue.cc:237] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.709542 4257 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.710628 4218 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.712693 4218 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.713881 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.705192 1996 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:40.706204 4276 raft_consensus.cc:2741] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 19:38:40.722347 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.724143 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:40.724601 4266 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.724614 4230 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:40.734289 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.738646 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.742278 2111 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:40.743754 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.757670 4269 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.741782 4260 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:40.740664 4276 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:40.740438 4266 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.742027 4257 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260927 19:38:40.747527 4174 consensus_queue.cc:1046] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.013s
I20260927 19:38:40.735841 3776 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:40.776758 4257 raft_consensus.cc:3032] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:40.750777 2037 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.779618 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:40.779970 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.781458 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.782450 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:40.783325 1996 raft_consensus.cc:2396] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 19:38:40.783452 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.777383 4257 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:40.785298 4266 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:40.788476 4205 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:40.788426 4174 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:40.790997 4261 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.795857 4276 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:40.808895 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:40.808208 4174 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:40.812695 4174 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:40.800905 4266 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:40.825945 2110 leader_election.cc:334] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.572s (SENT)
I20260927 19:38:40.822094 4230 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 19:38:40.831156 1961 leader_election.cc:334] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.573s (SENT)
I20260927 19:38:40.832080 1961 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:40.833992 2110 leader_election.cc:334] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.511s (SENT)
I20260927 19:38:40.802896 4260 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260927 19:38:40.822085 4266 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:40.814743 4205 raft_consensus.cc:686] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:40.852866 2688 raft_consensus.cc:1226] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:40.840296 4191 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260927 19:38:40.859476 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 60386.2us
I20260927 19:38:40.824766 4276 consensus_queue.cc:1046] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260927 19:38:40.845954 4257 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.862777 4208 consensus_queue.cc:1046] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260927 19:38:40.866602 4205 consensus_queue.cc:237] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:40.905416 1996 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 19:38:40.916646 4291 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20260927 19:38:40.918694 4292 log.cc:924] Time spent T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260927 19:38:40.921190 4218 raft_consensus.cc:3032] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:40.921828 4218 raft_consensus.cc:729] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:40.922152 4257 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:40.922854 4218 consensus_queue.cc:260] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:40.925400 4187 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260927 19:38:40.927251 1961 leader_election.cc:334] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.033s (SENT)
I20260927 19:38:40.930537 1961 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:40.931468 4048 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:40.934448 2143 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 19:38:40.982946 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 122630us
I20260927 19:38:40.945569 4261 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.037s
I20260927 19:38:40.960561 4218 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:40.957861 4294 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:41.014249 2686 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 19:38:41.031592 2140 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:41.033876 4063 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:41.038223 4269 raft_consensus.cc:3032] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:41.038678 4269 raft_consensus.cc:729] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:41.039494 4269 consensus_queue.cc:260] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.040716 4269 raft_consensus.cc:3037] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:41.002134 4294 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.053792 4294 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.057545 2143 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260927 19:38:41.059890 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 75473.2us
I20260927 19:38:41.060451 3971 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:41.062875 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.063755 2064 raft_consensus.cc:2379] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.063597 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.064430 2067 raft_consensus.cc:2354] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4.
I20260927 19:38:41.066026 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.066326 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.066792 2069 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:41.066691 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.070289 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.070664 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.002629 4295 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260927 19:38:41.083567 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.084573 2071 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.067875 2065 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:41.022032 4208 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260927 19:38:41.091662 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.093216 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.110865 2065 raft_consensus.cc:1226] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:41.104175 2069 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:41.134838 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.137202 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.137868 2065 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:41.140916 2069 raft_consensus.cc:1226] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260927 19:38:41.145478 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46711.1us
I20260927 19:38:41.158632 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.161409 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.162119 2062 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:41.173308 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.178247 2688 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260927 19:38:41.174451 2063 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:41.189074 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.190091 2068 raft_consensus.cc:2379] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.190757 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0f797e07d634f6abe29556a11863581"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:38:41.191286 2068 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:41.191676 2068 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:41.192309 2067 raft_consensus.cc:2396] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260927 19:38:41.196264 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.196374 3990 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:41.197168 2066 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:41.198511 2066 raft_consensus.cc:1226] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:41.136751 4174 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.176867 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.206849 2069 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
W20260927 19:38:41.217474 2110 leader_election.cc:334] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.791s (SENT)
W20260927 19:38:41.225775 2110 leader_election.cc:334] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.025s (SENT)
I20260927 19:38:41.227955 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.228770 2066 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.233086 2071 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:41.237908 2068 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.246742 2068 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.256242 1992 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:41.257347 2065 raft_consensus.cc:1226] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 19:38:41.259614 4174 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.261062 4266 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.274356 4187 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:41.280367 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:41.306342 2110 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.883s (SENT) [suppressed 59 similar messages]
I20260927 19:38:41.308039 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.314898 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.316129 2070 raft_consensus.cc:2396] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260927 19:38:41.319281 2653 leader_election.cc:334] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.711s (SENT)
W20260927 19:38:41.325840 1892 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 240) took 13716 ms (client timeout 15000 ms). Trace:
I20260927 19:38:41.328388 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:41.326321 1892 rpcz_store.cc:269] 0927 19:38:27.609695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:27.609809 (+ 114us) service_pool.cc:227] Handling call
0927 19:38:41.325776 (+13715967us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:40.626764 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:40.627505 (+ 741us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:40.627549 (+ 44us) write_op.cc:435] Acquired schema lock
0927 19:38:40.627573 (+ 24us) tablet.cc:660] Decoding operations
0927 19:38:40.628173 (+ 600us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:40.628263 (+ 90us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:40.628303 (+ 40us) tablet.cc:683] Acquiring locks for 2 operations
0927 19:38:40.672095 (+ 43792us) tablet.cc:699] Row locks acquired
0927 19:38:40.672116 (+ 21us) write_op.cc:260] PREPARE: finished
0927 19:38:40.672266 (+ 150us) write_op.cc:270] Start()
0927 19:38:40.672373 (+ 107us) write_op.cc:276] Timestamp: P: 1790537920672209 usec, L: 0
0927 19:38:40.672400 (+ 27us) op_driver.cc:331] REPLICATION: starting
0927 19:38:40.673032 (+ 632us) log.cc:842] Serialized 929 byte log entry
0927 19:38:40.808462 (+135430us) op_driver.cc:444] REPLICATION: finished
0927 19:38:41.020716 (+212254us) write_op.cc:301] APPLY: starting
0927 19:38:41.021997 (+ 1281us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:41.055131 (+ 33134us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:41.082171 (+ 27040us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:41.094092 (+ 11921us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:41.094290 (+ 198us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0927 19:38:41.094336 (+ 46us) write_op.cc:312] APPLY: finished
0927 19:38:41.094522 (+ 186us) log.cc:842] Serialized 29 byte log entry
0927 19:38:41.094711 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:41.094797 (+ 86us) write_op.cc:454] Released schema lock
0927 19:38:41.094837 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:41.095094 (+ 257us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:41.095125 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":188555,"apply.run_cpu_time_us":2741,"apply.run_wall_time_us":82948,"num_ops":2,"prepare.queue_time_us":99929,"prepare.run_cpu_time_us":3426,"prepare.run_wall_time_us":46577,"replication_time_us":132163,"thread_start_us":171553,"threads_started":2}]]}
I20260927 19:38:41.315037 4266 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:41.341526 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.344424 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.345875 2067 raft_consensus.cc:2379] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:41.349555 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.349421 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.352288 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.357225 1890 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.370362 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.371405 2062 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.371639 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.377518 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.367321 4261 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:41.374754 1890 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.380131 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.380952 4261 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.390627 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.388284 1890 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.391566 2070 raft_consensus.cc:2396] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260927 19:38:41.392484 4261 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.392304 1890 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 2 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.394174 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.394949 2069 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.398056 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.401033 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.401769 2068 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:41.403407 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.398845 2140 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:41.405206 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.405966 2068 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:41.406546 2654 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.407573 4261 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:41.407974 4261 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:41.408075 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.408349 4261 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:41.408900 2068 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:41.413959 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.415548 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.416402 2068 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:41.413690 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.419296 2069 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:41.419821 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.420737 2068 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.421674 1995 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:41.424254 4189 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:41.421613 2067 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.428399 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.429282 2062 raft_consensus.cc:1261] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260927 19:38:41.429687 2062 pending_rounds.cc:76] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 2
I20260927 19:38:41.429916 2062 pending_rounds.cc:97] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 19:38:41.394013 1890 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.430143 2062 raft_consensus.cc:2866] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:41.431104 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.431775 2063 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:41.431936 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.431684 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.433727 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.434095 2062 raft_consensus.cc:2379] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.434340 2065 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:41.434762 1890 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.440009 2065 raft_consensus.cc:3032] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:41.440105 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.440877 2063 raft_consensus.cc:3032] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:41.454335 2065 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:41.454639 2063 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:41.454790 2062 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:41.455248 2065 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.456326 2065 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:41.458767 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.442062 2066 raft_consensus.cc:3032] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:41.441931 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.440923 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.460135 2066 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:41.440464 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.442574 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.448849 4218 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260927 19:38:41.441188 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.463591 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.464519 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.467000 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.467136 2071 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.460901 2066 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.470417 2066 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:41.455423 2063 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.471218 2063 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:41.483184 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.488970 2065 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260927 19:38:41.493325 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.494601 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.494822 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.489323 2065 pending_rounds.cc:76] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 2
I20260927 19:38:41.496167 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.496701 2065 pending_rounds.cc:97] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260927 19:38:41.497371 4043 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:41.497938 4043 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.500787 4043 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.474260 4261 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.494743 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.502192 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.504688 2145 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:41.501165 2063 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:41.505846 2654 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.506737 4272 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:41.507135 4272 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:41.507462 4272 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:41.492671 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.517496 4261 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.517982 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.520282 2145 raft_consensus.cc:1203] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[5.6-5.6] Dedup: 5.6->[]
I20260927 19:38:41.521998 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.522255 2145 raft_consensus.cc:1226] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260927 19:38:41.520885 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.523375 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.524042 2142 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:41.521163 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.521653 2143 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260927 19:38:41.523447 2138 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:41.525851 4218 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:41.527740 2066 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260927 19:38:41.520860 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.530610 1992 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:41.531772 2066 pending_rounds.cc:76] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 2
I20260927 19:38:41.532639 2142 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:41.533663 4272 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.534409 2654 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.535240 4043 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:41.537201 4043 raft_consensus.cc:686] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:41.513337 2065 raft_consensus.cc:2866] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:41.538033 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.538586 2138 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:41.535808 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.535890 4272 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.532668 2066 pending_rounds.cc:97] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260927 19:38:41.542141 2066 raft_consensus.cc:2866] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:41.544500 2142 raft_consensus.cc:1203] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3] Dedup: 4.3->[]
I20260927 19:38:41.545010 2142 raft_consensus.cc:1226] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 19:38:41.545198 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.546710 4043 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.553076 2138 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:41.554662 2654 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.555567 4272 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:41.556169 4272 raft_consensus.cc:686] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:41.556984 4272 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.575289 4261 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:41.576385 2140 raft_consensus.cc:1203] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[4.4-4.4] Dedup: 4.4->[]
I20260927 19:38:41.576242 4261 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.576861 2140 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260927 19:38:41.575337 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.580981 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.581730 1987 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:41.582633 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.582713 2654 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:41.583818 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.584335 4150 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:41.584642 2142 raft_consensus.cc:2354] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:41.584832 4150 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:41.585596 4150 raft_consensus.cc:3037] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:41.585899 2138 raft_consensus.cc:1203] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20260927 19:38:41.586400 2138 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 19:38:41.584926 4261 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.588785 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.589924 4218 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:41.590481 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:41.590451 4218 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.592589 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:41.584854 2143 raft_consensus.cc:2379] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:41.594784 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.594564 4150 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.595398 2140 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:41.596688 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.597244 4150 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.598634 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:41.599547 1992 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:38:41.600028 4217 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:41.601765 4217 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:41.602445 4217 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:41.605512 4218 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.607195 1995 raft_consensus.cc:2454] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:41.610643 2654 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:41.611572 4150 raft_consensus.cc:2781] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:41.611964 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.612238 4150 raft_consensus.cc:686] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:41.612793 2143 raft_consensus.cc:2379] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:41.613574 2110 leader_election.cc:334] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.057s (SENT)
I20260927 19:38:41.613312 4150 consensus_queue.cc:237] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.618443 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.619323 2143 raft_consensus.cc:2379] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:41.619731 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:41.625756 2143 raft_consensus.cc:1203] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3] Dedup: 4.3->[]
I20260927 19:38:41.637831 2143 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 19:38:41.643720 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 155) took 6482 ms. Trace:
W20260927 19:38:41.645066 1935 catalog_manager.cc:5018] ChangeConfig:REMOVE_PEER RPC for tablet da05114b545f4ec9a25e86d9c059fea3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 6
I20260927 19:38:41.650089 1890 rpcz_store.cc:276] 0927 19:38:35.161893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:35.163024 (+ 1131us) service_pool.cc:227] Handling call
0927 19:38:41.643655 (+6480631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:41.531259 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:41.531759 (+ 500us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:41.531801 (+ 42us) write_op.cc:435] Acquired schema lock
0927 19:38:41.531818 (+ 17us) tablet.cc:660] Decoding operations
0927 19:38:41.545569 (+ 13751us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:41.545654 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:41.545680 (+ 26us) tablet.cc:683] Acquiring locks for 6 operations
0927 19:38:41.546226 (+ 546us) tablet.cc:699] Row locks acquired
0927 19:38:41.546247 (+ 21us) write_op.cc:260] PREPARE: finished
0927 19:38:41.546403 (+ 156us) write_op.cc:270] Start()
0927 19:38:41.546519 (+ 116us) write_op.cc:276] Timestamp: P: 1790537921546355 usec, L: 0
0927 19:38:41.546540 (+ 21us) op_driver.cc:331] REPLICATION: starting
0927 19:38:41.547174 (+ 634us) log.cc:842] Serialized 2574 byte log entry
0927 19:38:41.600638 (+ 53464us) op_driver.cc:444] REPLICATION: finished
0927 19:38:41.606156 (+ 5518us) write_op.cc:301] APPLY: starting
0927 19:38:41.606242 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:41.606430 (+ 188us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:41.606551 (+ 121us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:41.607332 (+ 781us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:41.607538 (+ 206us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0927 19:38:41.607585 (+ 47us) write_op.cc:312] APPLY: finished
0927 19:38:41.607832 (+ 247us) log.cc:842] Serialized 53 byte log entry
0927 19:38:41.617987 (+ 10155us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:41.618110 (+ 123us) write_op.cc:454] Released schema lock
0927 19:38:41.618159 (+ 49us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":86313,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":5236,"num_ops":6,"prepare.queue_time_us":35985,"prepare.run_cpu_time_us":3482,"prepare.run_wall_time_us":22384,"replication_time_us":53993,"thread_start_us":54401,"threads_started":2}]]}
I20260927 19:38:41.610098 4217 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.658707 2142 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:41.659685 2142 raft_consensus.cc:595] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Received request to transfer leadership
I20260927 19:38:41.673693 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:41.674657 2138 raft_consensus.cc:595] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Received request to transfer leadership
I20260927 19:38:41.672544 4272 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:41.676810 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.677640 1987 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:41.677429 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.679435 2142 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:41.683146 1987 raft_consensus.cc:1203] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20260927 19:38:41.677294 4272 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.684366 1987 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 19:38:41.686053 4272 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.679059 1891 catalog_manager.cc:5495] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.688869 1995 raft_consensus.cc:1203] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[5.3-5.3] Dedup: 5.3->[]
I20260927 19:38:41.689460 1995 raft_consensus.cc:1226] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260927 19:38:41.689524 1891 catalog_manager.cc:5495] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 4, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.696489 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.697100 1991 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:41.697433 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.695472 1891 catalog_manager.cc:5495] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.697106 1992 raft_consensus.cc:1203] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Deduplicated request from leader. Original: 3.3->[4.4-4.4] Dedup: 4.4->[]
I20260927 19:38:41.700227 1992 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260927 19:38:41.699596 1891 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:41.701485 4217 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.701678 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:41.702375 1991 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:38:41.702474 4189 consensus_queue.cc:1046] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:41.705835 1891 catalog_manager.cc:5495] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 5, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.706436 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.708250 2138 raft_consensus.cc:3032] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Stepping down as leader of term 1
I20260927 19:38:41.708606 2138 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:41.703275 4189 raft_consensus.cc:3032] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:41.707257 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:41.710389 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.710089 1891 catalog_manager.cc:5495] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 7, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:41.711174 2145 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:41.675607 4150 consensus_queue.cc:1035] T efcb4aec6b174f11b08d4483d9f63866 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260927 19:38:41.715126 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:41.716591 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.717435 2140 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:41.679044 4261 consensus_queue.cc:1046] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260927 19:38:41.718312 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:41.719075 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:41.719676 1991 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:41.720602 2138 consensus_queue.cc:260] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.698391 4187 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260927 19:38:41.719657 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:41.717243 1891 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 4, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.709636 4189 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:41.726266 4189 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:41.731685 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 62521.7us
I20260927 19:38:41.734961 2138 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 1 FOLLOWER]: Advancing to term 2
I20260927 19:38:41.737426 4261 raft_consensus.cc:3032] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:41.737673 4328 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:41.737262 2145 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:41.729066 1891 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 5, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.740266 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.741048 1891 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.725171 4272 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:41.703634 4186 consensus_queue.cc:1046] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20260927 19:38:41.748476 3981 consensus_queue.cc:1046] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260927 19:38:41.750427 3981 raft_consensus.cc:3032] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:41.750871 3981 raft_consensus.cc:729] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:41.747536 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.743197 4217 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:41.748521 4189 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:41.748726 4272 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:41.755195 4272 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
W20260927 19:38:41.750535 4321 log.cc:924] Time spent T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.106s user 0.003s sys 0.000s
I20260927 19:38:41.737941 4261 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:41.787593 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.789870 2064 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:41.746685 2138 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:41.781955 2068 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:41.783475 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.783980 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.784552 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.785030 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.749933 1891 catalog_manager.cc:5495] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.785491 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.786226 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.792208 2066 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
W20260927 19:38:41.788172 1961 leader_election.cc:334] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.545s (SENT)
I20260927 19:38:41.788833 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.794837 1961 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:41.800086 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.801751 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.802536 2070 raft_consensus.cc:2396] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:41.798059 1891 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 3 to 6, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.803984 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.807893 1991 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 2.
I20260927 19:38:41.804584 2069 raft_consensus.cc:2362] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3. Re-sending same reply.
I20260927 19:38:41.811744 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.803952 2065 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:41.820899 2062 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:41.821522 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.821977 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.822193 2067 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:41.823711 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.824271 2062 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:41.824426 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.787079 4217 raft_consensus.cc:686] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:41.824496 2063 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:41.826685 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.828681 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.826593 3981 consensus_queue.cc:260] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.829996 4217 consensus_queue.cc:237] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.830650 3981 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:41.831408 4325 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:41.831950 4325 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:41.827119 1891 catalog_manager.cc:5495] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.834590 1891 catalog_manager.cc:5495] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.836256 1891 catalog_manager.cc:5495] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.838158 2062 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:41.838246 1891 catalog_manager.cc:5495] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:41.839588 2063 raft_consensus.cc:2362] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5. Re-sending same reply.
I20260927 19:38:41.812933 4048 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:41.845283 1891 catalog_manager.cc:5495] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
W20260927 19:38:41.845943 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 112863us
I20260927 19:38:41.846645 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.847465 2069 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:41.847101 1891 catalog_manager.cc:5495] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.906652 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.906207 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.847735 2653 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:41.887992 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:41.913877 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 66204us
I20260927 19:38:41.867233 2067 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:41.890182 2065 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260927 19:38:41.918565 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 47799.4us
W20260927 19:38:41.901618 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 119673us
I20260927 19:38:41.933565 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.918048 1891 catalog_manager.cc:5495] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 2, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 2 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:41.934475 2110 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:41.934804 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:41.935166 2070 raft_consensus.cc:2362] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6. Re-sending same reply.
I20260927 19:38:41.935488 2138 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:41.914403 2062 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:41.938498 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.939436 2063 raft_consensus.cc:2362] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5. Re-sending same reply.
I20260927 19:38:41.935902 1891 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.935863 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:41.904841 4331 raft_consensus.cc:979] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:41.940449 1891 catalog_manager.cc:5495] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.947824 1891 catalog_manager.cc:5495] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 4, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.904292 4260 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:41.913242 4174 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:41.872211 4325 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:41.953649 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.954684 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:41.878911 3776 raft_consensus.cc:3032] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:41.926648 4328 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:41.951880 2654 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:41.949275 1891 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:41.970089 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.969431 2062 raft_consensus.cc:2362] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5. Re-sending same reply.
W20260927 19:38:41.980110 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 44334.6us
I20260927 19:38:41.934221 4043 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260927 19:38:41.990170 2065 raft_consensus.cc:3032] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:41.986536 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.990897 2065 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:41.991582 2065 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:41.992027 2110 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:41.992538 2065 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:41.985874 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.986157 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:41.981307 1891 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.994976 1891 catalog_manager.cc:5495] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.996621 1891 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:42.001775 2065 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260927 19:38:42.002031 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.002377 2065 pending_rounds.cc:76] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 3
I20260927 19:38:42.002861 2065 pending_rounds.cc:97] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260927 19:38:42.002853 2064 raft_consensus.cc:2362] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3. Re-sending same reply.
I20260927 19:38:42.004676 2062 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 19:38:42.004673 2064 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260927 19:38:42.005481 2065 raft_consensus.cc:2866] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:42.007187 2069 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:42.007526 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:42.008494 2065 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:42.008383 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.011458 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:42.011147 1891 catalog_manager.cc:5495] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:42.012264 2067 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:38:41.969852 4164 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:42.013764 1891 catalog_manager.cc:5495] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:42.017050 2110 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:41.848408 4261 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.017812 1891 catalog_manager.cc:5495] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.976233 4328 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.041756 2068 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260927 19:38:42.042372 1961 leader_election.cc:334] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.728s (SENT)
I20260927 19:38:42.042488 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:42.042837 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:41.972124 3776 raft_consensus.cc:729] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:42.044049 4333 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:42.038504 1891 catalog_manager.cc:5495] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:42.044809 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.045934 2063 raft_consensus.cc:2396] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:42.020349 2110 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:41.983126 2679 raft_consensus.cc:1226] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:42.045470 1891 catalog_manager.cc:5495] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:41.970749 4174 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.018731 4339 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:42.020655 4164 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.048293 1891 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 2 to 3, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 3 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:42.113169 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:42.118489 2068 raft_consensus.cc:2362] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3. Re-sending same reply.
I20260927 19:38:42.040946 4189 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 19:38:42.114248 1961 leader_election.cc:334] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.781s (SENT)
I20260927 19:38:42.060060 4328 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:42.118979 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.120821 1961 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:41.970209 4260 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.123152 2065 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:42.018924 4205 raft_consensus.cc:1067] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:42.152408 2068 raft_consensus.cc:1261] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260927 19:38:42.173744 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 59531.5us
I20260927 19:38:42.173823 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:42.174520 1961 leader_election.cc:334] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.798s (SENT)
I20260927 19:38:42.069092 4344 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:42.174642 2069 raft_consensus.cc:1261] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:42.177011 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.150738 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:42.181917 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 161646us
W20260927 19:38:42.181999 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 118290us
I20260927 19:38:42.182919 2110 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:42.189348 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:42.190099 2065 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:42.191646 1961 leader_election.cc:334] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.923s (SENT)
I20260927 19:38:42.180996 2063 raft_consensus.cc:2362] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7. Re-sending same reply.
I20260927 19:38:42.179901 2064 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:42.211390 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.149569 4164 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:42.120707 4189 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:42.215381 2064 raft_consensus.cc:3032] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:42.213799 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:42.233271 2064 raft_consensus.cc:729] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:42.233489 2063 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
W20260927 19:38:42.213846 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 153455us
I20260927 19:38:42.229571 2070 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:42.229214 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:42.230638 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:42.235349 2069 raft_consensus.cc:2362] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5. Re-sending same reply.
I20260927 19:38:42.236140 2068 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:42.234061 2064 consensus_queue.cc:260] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.239225 2064 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:42.320474 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 135550us
I20260927 19:38:42.229137 2065 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:42.190150 4217 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:42.189484 4344 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:42.340749 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.342538 2110 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:42.115209 4272 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.349438 2110 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:42.371743 2064 raft_consensus.cc:1261] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:42.242053 4189 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:42.393777 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 174894us
I20260927 19:38:42.397310 1961 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.176056 3776 consensus_queue.cc:260] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.361233 4344 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:42.359400 4205 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:42.423861 2068 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:42.413522 4343 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:42.432221 4205 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.433465 4205 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:42.415720 3990 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:42.416491 4339 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 19:38:42.458048 4346 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20260927 19:38:42.459962 4334 log.cc:924] Time spent T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.335s user 0.001s sys 0.000s
W20260927 19:38:42.461802 4336 log.cc:924] Time spent T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.282s user 0.000s sys 0.001s
I20260927 19:38:42.463040 4325 consensus_queue.cc:1035] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:42.440641 4339 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.464847 4339 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:42.470778 4347 consensus_queue.cc:1035] T ba36610baf2046bc9b0bd15df9129f16 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:42.482612 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:42.437683 4333 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:38:42.488528 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 91304.5us
I20260927 19:38:42.430231 4355 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:42.532902 1961 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:42.535547 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 194872us
I20260927 19:38:42.475174 3776 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:42.535908 2069 raft_consensus.cc:2932] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:42.550379 4357 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260927 19:38:42.572144 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.570073 4355 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:42.578205 4359 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:42.579031 4359 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:42.579577 4359 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.480118 4174 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:42.606333 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:42.604301 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 49256us
I20260927 19:38:42.490751 4338 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:42.447643 3990 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.608466 2145 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 19:38:42.622814 2109 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.009s sys 0.002s
W20260927 19:38:42.623216 2109 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.009s sys 0.002s
W20260927 19:38:42.632788 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 282587us
W20260927 19:38:42.634119 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44357.5us
I20260927 19:38:42.633577 2110 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:42.642460 2145 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:42.609884 4174 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:42.623659 3981 consensus_meta.cc:287] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.793s user 0.012s sys 0.000s
I20260927 19:38:42.487120 4350 consensus_queue.cc:1035] T 069da74585d54798b7b8d625c46ae785 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.145s
I20260927 19:38:42.609656 4338 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.649987 2145 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 19:38:42.652848 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.653705 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:42.658810 2138 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:42.608966 4333 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260927 19:38:42.632647 4272 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:42.650943 4361 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:42.658981 4332 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:42.610687 4356 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260927 19:38:42.675781 3990 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:42.676052 4333 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:42.676981 4341 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.677224 4186 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 19:38:42.679152 2108 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.001s sys 0.012s
I20260927 19:38:42.619403 4363 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260927 19:38:42.684332 4355 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:42.667662 4260 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:42.662914 4338 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:42.692490 1995 raft_consensus.cc:1261] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260927 19:38:42.703836 2108 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.889s user 0.002s sys 0.013s
W20260927 19:38:42.716073 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 901603us
I20260927 19:38:42.679766 4344 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.706740 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:42.658486 4174 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.703253 4360 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:42.709424 4332 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:42.710649 4189 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:42.722600 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 189747us
I20260927 19:38:42.702847 4333 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:42.685441 4354 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:42.749162 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.696630 4272 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:42.753187 1992 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:38:42.713042 4186 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.762068 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.744644 4353 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:42.746601 4260 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:42.722649 4150 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
W20260927 19:38:42.803706 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 150956us
I20260927 19:38:42.738785 4335 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260927 19:38:42.803606 4364 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:42.806990 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.763949 4186 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:42.811728 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:42.814286 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 180779us
I20260927 19:38:42.761633 4371 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:42.815191 2110 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:42.687868 4361 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.749691 4332 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.723809 4341 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.826720 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:42.832887 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:42.751256 4360 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.745663 4369 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.102s
I20260927 19:38:42.806334 4375 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:42.804018 4367 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:42.843501 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:42.762431 4272 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.812963 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:42.841735 4325 consensus_queue.cc:1035] T ac63cb1bcdd141bb9de9fcb5d0f102af P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260927 19:38:42.838624 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 77603.5us
I20260927 19:38:42.816869 4048 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:42.812180 4378 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:42.851414 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:42.819753 4359 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:42.852823 4186 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.792861 4339 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.853004 4327 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:42.831423 4260 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:42.867009 2141 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:42.757699 4354 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.900677 2140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:42.937824 2140 raft_consensus.cc:595] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Received request to transfer leadership
I20260927 19:38:42.902252 2139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:42.938910 2139 raft_consensus.cc:595] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:42.939373 2139 raft_consensus.cc:603] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 19:38:42.901778 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 47255.4us
W20260927 19:38:42.905505 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 93877.4us
I20260927 19:38:42.941490 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.835036 4371 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:42.950310 4358 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260927 19:38:42.951119 4186 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:42.822660 4363 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260927 19:38:42.951047 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:42.975756 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 171622us
I20260927 19:38:42.977474 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:42.979352 2037 leader_election.cc:398] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:42.979744 2037 leader_election.cc:401] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 19:38:42.817938 4205 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.953258 2138 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:42.834122 4377 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:42.984656 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:43.036681 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 52120.4us
I20260927 19:38:42.809330 4353 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:42.952721 4187 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:42.952036 4373 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:42.983860 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.039552 2145 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:42.909035 4375 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260927 19:38:43.052068 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 290056us
I20260927 19:38:42.934244 4327 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:43.034749 4331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.061s user 0.025s sys 0.086s
W20260927 19:38:42.908625 4164 consensus_meta.cc:287] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.571s user 0.011s sys 0.001s
W20260927 19:38:43.051433 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 207988us
I20260927 19:38:43.071702 2136 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:43.039659 4186 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.036708 4383 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:42.809016 4333 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:43.057531 1942 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 228a1cf257434cbe9e5616dbd94e1864 already in progress
I20260927 19:38:42.909246 4378 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:43.054497 1961 leader_election.cc:334] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.502s (SENT)
I20260927 19:38:42.810081 4364 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:42.953272 4260 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.045987 4384 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:43.061815 4331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.088s user 0.025s sys 0.087s
I20260927 19:38:43.083659 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:43.210958 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 156467us
I20260927 19:38:43.064723 4150 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:43.100143 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 233977us
I20260927 19:38:42.936671 4372 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:43.046002 4381 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:43.197570 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 218424us
I20260927 19:38:43.218443 4384 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.079108 4174 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.265632 4381 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.266508 4174 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.267004 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.267542 4384 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.267031 4174 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.268021 4384 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.268453 4384 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.268695 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.054301 4344 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.058615 4341 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.269781 4174 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.059087 4327 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:43.079325 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 228a1cf257434cbe9e5616dbd94e1864 already in progress
I20260927 19:38:42.981338 4359 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260927 19:38:42.855602 4367 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.079612 4186 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.268702 4381 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.072227 4189 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.080677 2139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
W20260927 19:38:43.255569 4343 consensus_meta.cc:287] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.804s user 0.006s sys 0.003s
W20260927 19:38:43.198518 4358 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260927 19:38:43.075513 4383 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:43.075780 4375 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:43.297999 4363 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:43.386860 4363 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.270567 4174 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:43.389577 4174 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:43.390074 4174 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:43.315383 4186 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.335858 4402 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:43.394040 4402 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.317714 4364 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.322079 4332 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.323161 4354 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:43.324631 4378 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.395723 4378 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:43.396194 4378 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.394040 4363 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.399128 4384 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.399698 4384 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:43.070839 4164 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.335687 4359 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:43.401006 4379 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.401676 4379 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.402141 4164 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.401929 4174 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.403082 2143 raft_consensus.cc:1226] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:43.403735 4379 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.336537 4380 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:43.335726 4385 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260927 19:38:43.336930 4376 log.cc:924] Time spent T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.325s user 0.000s sys 0.001s
I20260927 19:38:43.337415 4391 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.335757 4394 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.134s
I20260927 19:38:43.338018 4386 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.119s
I20260927 19:38:43.335827 4388 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:43.338753 4381 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 19:38:43.335788 4387 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:43.335671 4382 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:43.340088 4361 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.341544 2680 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:43.343387 4389 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:43.344427 4360 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.210660 3981 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.131s
I20260927 19:38:43.263350 4150 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.350381 4353 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.354470 4339 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.284233 4335 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:38:43.356828 4272 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.370867 4367 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.400950 4359 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:43.540401 4359 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.541452 4359 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:43.542032 4327 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.543047 2139 raft_consensus.cc:595] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:43.543494 2139 raft_consensus.cc:603] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:43.544655 4261 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:43.336067 4396 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:43.546806 4150 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.548305 4335 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:43.546806 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.540426 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.263063 4260 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.382884 4343 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260927 19:38:43.405040 4328 consensus_meta.cc:287] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 1.226s user 0.012s sys 0.000s
I20260927 19:38:43.518225 4205 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.396194 4402 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.406503 4378 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.400133 4384 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.437510 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:43.556802 2141 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:42.984437 4371 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:43.440496 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 457496us
I20260927 19:38:43.559875 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.440706 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 1941) took 1448 ms. Trace:
I20260927 19:38:43.406610 4174 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.432737 4383 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:43.502919 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.570911 2687 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:43.504063 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.571475 4384 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.573462 2108 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:43.505911 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 145483us
I20260927 19:38:43.574584 4339 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:43.574940 4339 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:43.575269 4339 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:43.575070 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.575985 2140 raft_consensus.cc:2379] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:38:43.508999 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 240360us
I20260927 19:38:43.576776 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.577582 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.578706 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:43.463850 4394 consensus_queue.cc:1225] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer 4cc2df048f424e08803a390946004df0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260927 19:38:43.581725 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:43.558614 4328 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.582389 2066 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:43.583364 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:43.583885 2066 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:43.584127 4328 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.584450 2686 raft_consensus.cc:1226] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:43.554652 4393 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.450973 4380 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.585695 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:43.585953 2686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 667) took 1457 ms. Trace:
I20260927 19:38:43.586510 2067 raft_consensus.cc:2379] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:43.586378 2686 rpcz_store.cc:276] 0927 19:38:42.128746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:42.173598 (+ 44852us) service_pool.cc:227] Handling call
0927 19:38:42.173869 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:43.584863 (+1410994us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:43.585723 (+ 860us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54006ed054. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:43.585896 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3243381120}
I20260927 19:38:43.587366 4380 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.587551 2067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:38:43.588088 2067 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:43.588428 2067 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:43.589298 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:43.590032 2068 raft_consensus.cc:2354] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6.
I20260927 19:38:43.591732 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.502063 4164 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:43.502599 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.593811 2679 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:43.595009 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.504693 4379 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.597905 4379 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.549244 4335 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.599932 4379 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.529443 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.534447 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.512046 4381 raft_consensus.cc:686] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:43.601099 1989 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:43.601447 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 265) took 1411 ms. Trace:
I20260927 19:38:43.601548 4381 consensus_queue.cc:237] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.602660 4164 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.604760 4164 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.604857 4343 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:43.605455 4343 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.606830 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.585059 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:43.607421 4343 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:43.419524 3776 consensus_meta.cc:287] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.817s user 0.007s sys 0.003s
I20260927 19:38:43.608048 2071 raft_consensus.cc:2396] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 19:38:43.608497 4327 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:43.516633 4310 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:43.513960 4401 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.560629 2062 rpcz_store.cc:276] 0927 19:38:41.992087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:42.016466 (+ 24379us) service_pool.cc:227] Handling call
0927 19:38:42.016765 (+ 299us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:42.193063 (+176298us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:42.193115 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:42.193931 (+ 816us) log.cc:842] Serialized 26 byte log entry
0927 19:38:42.422580 (+228649us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:42.669738 (+247158us) log.cc:842] Serialized 14 byte log entry
0927 19:38:43.064025 (+394287us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b18003e7be0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f79135f182c 00007f7910a1c9dc 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a37155 00007f791395a3d0 00007f791396c576 00007f791397d0d7
0927 19:38:43.064162 (+ 137us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:43.064246 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:43.439075 (+374829us) raft_consensus.cc:1671] finished
0927 19:38:43.440483 (+ 1408us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:43.440635 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15288,"prepare.queue_time_us":51274,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":509556,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":24305,"spinlock_wait_cycles":3221376,"thread_start_us":628368,"threads_started":4,"wal-append.queue_time_us":610648}
I20260927 19:38:43.612447 2141 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:43.563802 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.567034 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.554616 4357 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:43.568264 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 264) took 1389 ms. Trace:
I20260927 19:38:43.569108 4371 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.510421 4388 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:43.569751 2651 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.002s sys 0.010s
I20260927 19:38:43.520280 4389 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.569947 4333 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:43.509449 4387 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.639309 2070 raft_consensus.cc:2396] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:43.528144 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.640283 1987 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:43.550849 1996 raft_consensus.cc:2396] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 19:38:43.595829 2143 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:43.607450 2688 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
W20260927 19:38:43.612740 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 47741.8us
I20260927 19:38:43.601939 2069 rpcz_store.cc:276] 0927 19:38:42.189948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:42.335916 (+145968us) service_pool.cc:227] Handling call
0927 19:38:42.442936 (+107020us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:42.443260 (+ 324us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:42.443287 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:42.456117 (+ 12830us) log.cc:842] Serialized 501 byte log entry
0927 19:38:42.535239 (+ 79122us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:42.641166 (+105927us) log.cc:842] Serialized 14 byte log entry
0927 19:38:43.600267 (+959101us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b54003bd420. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138f5fb3 00007f7913962f0f 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c
0927 19:38:43.600322 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:43.600400 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:43.600442 (+ 42us) raft_consensus.cc:1671] finished
0927 19:38:43.600527 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:43.601258 (+ 731us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400324450. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:43.601388 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2669,"raft.queue_time_us":113497,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":34613,"spinlock_wait_cycles":261601792,"thread_start_us":136989,"threads_started":2,"wal-append.queue_time_us":169744}
I20260927 19:38:43.615345 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.532305 4360 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:43.617238 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:43.642469 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:43.643059 2062 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:43.643949 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:43.632457 2651 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.003s sys 0.011s
I20260927 19:38:43.645001 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:43.645619 2062 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:43.637050 2067 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.620195 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.624972 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.634516 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.622448 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:43.634811 2063 rpcz_store.cc:276] 0927 19:38:42.178804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:42.234115 (+ 55311us) service_pool.cc:227] Handling call
0927 19:38:42.234438 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:42.425312 (+190874us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:42.425376 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:42.442619 (+ 17243us) log.cc:842] Serialized 26 byte log entry
0927 19:38:42.757388 (+314769us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:42.798782 (+ 41394us) log.cc:842] Serialized 14 byte log entry
0927 19:38:42.808658 (+ 9876us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:42.808742 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:43.568061 (+759319us) raft_consensus.cc:1671] finished
0927 19:38:43.568084 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:43.568207 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61167,"prepare.queue_time_us":39343,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":410046,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":82967,"thread_start_us":292123,"threads_started":2,"wal-append.queue_time_us":370990}
I20260927 19:38:43.638161 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:43.589320 4338 consensus_meta.cc:287] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.842s user 0.011s sys 0.000s
I20260927 19:38:43.462054 4391 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.650317 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.676749 1987 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:43.650905 2068 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:43.653996 2140 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:43.656304 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:43.652081 2654 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.694485 2140 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260927 19:38:43.700124 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 80017.8us
I20260927 19:38:43.701025 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.656559 2141 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:43.701067 1962 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.702971 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.690373 2063 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:43.703766 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.704226 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:43.704687 2686 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:43.710263 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.616427 4363 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:43.732285 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:43.732975 2110 leader_election.cc:334] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.016s (SENT)
I20260927 19:38:43.710273 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.711064 1990 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:38:43.714154 1959 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:43.730983 2067 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.731031 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.712598 2111 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.633524 4380 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.764240 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:43.766849 2143 raft_consensus.cc:595] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:43.733474 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:43.766304 1991 raft_consensus.cc:1799] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Handling vote request from an unknown peer 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:43.768977 1991 raft_consensus.cc:2396] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260927 19:38:43.731573 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:43.775570 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 61517.8us
W20260927 19:38:43.746608 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 94631.6us
I20260927 19:38:43.778245 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.778939 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:43.780431 2110 leader_election.cc:334] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.103s (SENT)
W20260927 19:38:43.781635 2110 leader_election.cc:334] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.115s (SENT)
W20260927 19:38:43.782567 2110 leader_election.cc:334] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.179s (SENT)
I20260927 19:38:43.783102 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.785615 4378 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:43.788393 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:43.795481 2653 leader_election.cc:334] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.199s (SENT)
I20260927 19:38:43.808974 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:43.795403 2066 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
W20260927 19:38:43.811102 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 98573.6us
I20260927 19:38:43.812386 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.730988 4359 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:43.711331 4384 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 19:38:43.732453 4395 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:43.646824 4218 consensus_queue.cc:1046] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.146s
I20260927 19:38:43.647579 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.640317 4217 consensus_queue.cc:1046] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.177s
I20260927 19:38:43.818315 2111 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.699749 4338 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.768220 4356 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:43.768522 4272 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:43.822196 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:43.880870 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 58739.3us
I20260927 19:38:43.881846 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:43.639186 4408 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.640563 4332 consensus_queue.cc:1046] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.180s
I20260927 19:38:43.806253 4396 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:43.806757 4260 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:43.922144 4378 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:43.922569 4408 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.632043 4401 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.927575 4332 raft_consensus.cc:3032] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:43.927989 4332 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:43.929102 4401 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.928754 4332 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:43.929253 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 110997us
I20260927 19:38:43.929961 4332 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:43.930248 2111 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.931164 4383 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.701611 4357 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.585079 4393 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.632261 4310 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.821240 4410 raft_consensus.cc:2741] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260927 19:38:43.763675 4364 consensus_queue.cc:1046] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260927 19:38:43.821530 4404 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:43.786878 4403 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 19:38:43.813819 2110 leader_election.cc:334] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.117s (SENT)
I20260927 19:38:43.812400 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:43.827875 3776 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:43.821831 4394 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:43.821915 4174 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:43.770241 4363 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.686081 4360 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.886631 2037 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:43.907624 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.950358 1996 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:43.908108 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:43.933246 4383 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.779781 2686 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:43.884836 4396 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:43.819722 4327 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:43.964721 4327 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:43.965242 4327 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:43.934051 2137 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:43.807178 4362 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
I20260927 19:38:43.872448 4356 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:43.925166 4408 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.943476 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:43.973158 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 159352us
I20260927 19:38:43.812690 4380 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:43.925659 4359 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:43.885109 4330 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:43.947500 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 135155us
I20260927 19:38:43.908999 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:43.936748 4218 raft_consensus.cc:3032] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:43.932740 4388 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:43.937551 4353 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.939975 4373 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:43.935783 4412 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.936095 4272 raft_consensus.cc:686] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:43.949584 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:43.958664 4396 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:43.963371 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 76831.5us
I20260927 19:38:44.172216 4363 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.946684 4401 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:43.973799 2069 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:43.982475 2654 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:43.961386 4347 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:43.972235 4356 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.121063 4381 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.049166 4330 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:43.977034 4218 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:43.968464 4174 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:43.977696 3981 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:43.901693 4339 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.098702 4371 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:44.054072 2686 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
W20260927 19:38:44.158663 4375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.014s sys 0.030s
W20260927 19:38:44.167526 4335 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.028s
I20260927 19:38:44.126129 4386 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.168865 4391 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:43.940321 4379 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:44.187970 4337 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:44.210582 4412 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.194456 4373 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.211836 4394 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.269857 4360 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.227494 4401 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.280278 4218 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.285820 4401 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.280869 4337 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.236586 4357 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.218979 4389 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.222630 4363 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:44.193862 4353 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:44.241178 4420 log.cc:924] Time spent T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260927 19:38:44.218943 4405 consensus_queue.cc:1035] T 4fe684e9563d491b9d4b3b99204450c8 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260927 19:38:44.226985 4387 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:44.277916 4416 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.151s user 0.002s sys 0.000s
I20260927 19:38:44.279722 4412 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.240360 4381 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.301743 4381 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:44.248147 4335 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.000s sys 0.028s
I20260927 19:38:44.301656 4356 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.222182 4347 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.303023 4347 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.303236 4392 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:44.303997 4356 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.303792 4392 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.305830 4392 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.279146 4330 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.306238 4395 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.306696 4395 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.283028 4394 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.309463 4396 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.310812 4394 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:44.282411 4414 log.cc:924] Time spent T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20260927 19:38:44.310734 4347 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.311611 4396 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.288707 4339 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.312891 4347 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:44.313624 4333 consensus_meta.cc:287] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.666s user 0.003s sys 0.008s
I20260927 19:38:44.314667 4353 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.283362 4380 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.274382 4386 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.289613 4338 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.290091 4310 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.242517 4393 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.316993 4386 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.282785 4373 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.317595 4327 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.292990 4400 raft_consensus.cc:3032] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:44.299294 4405 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:44.319607 4400 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:44.296396 4363 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.323649 4337 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.326788 4400 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.328500 4400 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:44.328531 4363 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.339944 4373 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.343189 4373 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.347882 4395 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:44.375962 1959 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.007s sys 0.032s
W20260927 19:38:44.377125 1959 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.590s user 0.008s sys 0.032s
W20260927 19:38:44.377732 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 590725us
W20260927 19:38:44.385699 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.040s sys 0.097s
W20260927 19:38:44.386370 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.040s sys 0.097s
W20260927 19:38:44.386888 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 598988us
I20260927 19:38:44.054382 2062 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:44.391006 2069 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:44.393105 2139 raft_consensus.cc:2354] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:44.215359 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.394639 2138 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:44.209913 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.398437 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:44.286890 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 304499us
I20260927 19:38:44.408344 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.289680 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.413609 2136 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2175) took 1564 ms. Trace:
I20260927 19:38:44.414395 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.290715 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.418061 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.420655 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2179) took 1569 ms. Trace:
I20260927 19:38:44.421168 2137 rpcz_store.cc:276] 0927 19:38:42.851350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:42.852761 (+ 1411us) service_pool.cc:227] Handling call
0927 19:38:42.853014 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:43.968140 (+1115126us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:44.420202 (+452062us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400142154. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:44.420470 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2485709696}
I20260927 19:38:44.422628 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.423182 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.423782 1990 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:38:44.425053 2144 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:44.419344 2142 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:44.412981 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.427984 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.432933 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.435510 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.437388 1996 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:44.439059 2685 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:44.290501 4272 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.289580 4218 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:44.441773 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.408980 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.473600 4383 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.473973 2141 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:44.474529 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.474481 4383 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.475885 1990 raft_consensus.cc:2396] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260927 19:38:44.454941 4327 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.457924 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.458431 4338 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:44.478049 4338 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.480764 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.485037 2685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34940 (request call id 430) took 1015 ms. Trace:
I20260927 19:38:44.467348 4353 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:44.485785 4353 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:44.486522 4353 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.473965 1991 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:44.469094 4395 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.464578 4384 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:44.485617 2685 rpcz_store.cc:276] 0927 19:38:43.448206 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:43.448385 (+ 179us) service_pool.cc:227] Handling call
0927 19:38:44.437830 (+989445us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:38:44.438442 (+ 612us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:44.446055 (+ 7613us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:44.463201 (+ 17146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10572160}
I20260927 19:38:44.414170 2136 rpcz_store.cc:276] 0927 19:38:42.848901 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:42.849022 (+ 121us) service_pool.cc:227] Handling call
0927 19:38:42.849418 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:43.078104 (+228686us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:44.412861 (+1334757us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400140d54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:44.413339 (+ 478us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":510561408}
I20260927 19:38:44.488580 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.488652 4384 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.481245 4338 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.501194 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.504376 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.506439 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.506304 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.509259 4335 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:44.503880 4384 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.511035 4335 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.516916 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.516997 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.518923 1991 raft_consensus.cc:3037] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.519234 4335 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.511821 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.521281 1988 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 2.
I20260927 19:38:44.529937 4395 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:44.527410 4383 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.539106 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1556) took 1016 ms. Trace:
I20260927 19:38:44.539561 1992 rpcz_store.cc:276] 0927 19:38:43.523093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:43.523209 (+ 116us) service_pool.cc:227] Handling call
0927 19:38:43.523494 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:43.710618 (+187124us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:43.710650 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:43.710689 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:43.710718 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:44.538615 (+827897us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54001ed2d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:44.538680 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:44.538905 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":429906688}
I20260927 19:38:44.541484 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.543598 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.547017 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.551230 2111 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.552090 4217 raft_consensus.cc:3032] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:44.552649 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.554749 4164 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:44.555276 4164 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.555655 4164 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.557716 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0f797e07d634f6abe29556a11863581"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:44.559825 2143 raft_consensus.cc:595] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:44.560585 2143 raft_consensus.cc:603] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:44.560897 2654 leader_election.cc:398] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:44.554639 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.561333 2654 leader_election.cc:401] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:44.566833 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.584062 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.584486 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.575078 2142 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:44.585495 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.552667 4217 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:44.586166 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.586727 2140 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.586845 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.587509 4217 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.588775 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.589903 2687 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:44.593103 2111 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.593966 2138 raft_consensus.cc:1226] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:44.594173 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:44.594626 2111 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.595273 2680 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:44.585582 2688 raft_consensus.cc:1226] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:44.587540 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.589087 4217 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:44.585691 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.629062 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.629231 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.632929 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.634279 2139 raft_consensus.cc:3037] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:44.633862 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 73368us
I20260927 19:38:44.635833 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.630733 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.637663 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.640822 4048 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.641237 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.650166 4355 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:38:44.650260 1995 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:44.650774 4355 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.651182 4355 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:38:44.653427 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.654520 4205 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:44.654964 4205 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.655282 4205 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.639675 1962 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.638797 4395 consensus_queue.cc:1046] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260927 19:38:44.642590 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.648988 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:44.659359 1991 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
W20260927 19:38:44.647061 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 74193.1us
I20260927 19:38:44.641595 4337 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 6
I20260927 19:38:44.664717 4363 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.665499 4363 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.667222 4164 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.667805 4363 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.668453 2143 raft_consensus.cc:2354] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:38:44.668663 4383 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.648291 3776 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:44.669224 4383 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.670415 2136 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.670054 4164 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.671000 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.671515 4383 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.672230 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.673059 2145 raft_consensus.cc:2379] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:44.669629 3776 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.673560 3776 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:44.677837 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.679399 4395 raft_consensus.cc:3032] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:44.679841 4395 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:44.680058 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.680311 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.681118 1987 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:44.682219 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.683329 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.682331 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.685258 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.685817 1994 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:44.685313 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.687249 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.689530 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.689828 3990 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:44.690208 1990 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.642079 4048 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:44.689553 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 85145.3us
I20260927 19:38:44.694144 4048 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.694526 1962 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.660138 2138 raft_consensus.cc:595] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:44.690307 3990 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:44.705315 3990 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:44.701056 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:44.706410 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:44.709298 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:44.710260 2682 raft_consensus.cc:2354] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6.
I20260927 19:38:44.711934 2139 raft_consensus.cc:2454] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:44.715975 4353 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.700322 4208 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:44.700101 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.700966 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.702064 4335 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:44.702913 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.729635 4335 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.730466 1994 raft_consensus.cc:2354] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6.
I20260927 19:38:44.703061 2108 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:44.704401 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.731815 2142 raft_consensus.cc:2379] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:44.680964 4395 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.697886 1995 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:44.724020 2140 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:44.728261 4208 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:44.732591 1990 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:38:44.734270 4335 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.735625 2136 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:44.735323 4208 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:44.738909 4353 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.687741 3776 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.739719 2680 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.750123 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.755436 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.757265 4339 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:44.760650 4364 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:44.761375 4364 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.763331 4395 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:44.762740 1962 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.764396 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.766899 4402 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.763167 4218 consensus_queue.cc:1046] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260927 19:38:44.769935 4363 consensus_queue.cc:1046] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:44.772440 4363 raft_consensus.cc:3032] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:44.774099 4363 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:44.759155 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.774748 4332 raft_consensus.cc:3032] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:44.775758 1989 raft_consensus.cc:2354] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:38:44.776036 4332 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:44.776952 4332 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.778891 4332 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:44.781708 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 79214.4us
I20260927 19:38:44.781834 4363 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.783488 4363 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:44.787801 2110 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:44.790083 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.790897 4337 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:44.797742 4339 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:44.799638 2687 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:44.799893 4339 consensus_queue.cc:1046] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:44.801729 4339 raft_consensus.cc:3032] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:38:44.802098 4339 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:44.802371 1962 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:44.803519 4362 raft_consensus.cc:2781] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:44.803038 4339 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.804050 4362 raft_consensus.cc:686] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:44.805058 4362 consensus_queue.cc:237] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.806430 4355 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.806763 3776 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:44.808593 4355 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.799674 2108 leader_election.cc:398] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6.
I20260927 19:38:44.809751 2108 leader_election.cc:401] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 19:38:44.809885 4208 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.811738 4394 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6.
I20260927 19:38:44.815042 2136 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:44.816133 4402 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:44.817734 4364 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.821799 2110 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:44.821923 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.822973 1989 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:44.825577 3776 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:44.827616 4208 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:44.834543 2143 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 8. Current term is 9. Ops: []
I20260927 19:38:44.837996 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:44.852022 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 49757.6us
I20260927 19:38:44.852075 2654 leader_election.cc:398] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:38:44.853035 2654 leader_election.cc:401] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260927 19:38:44.856050 1962 leader_election.cc:398] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:38:44.856642 1962 leader_election.cc:401] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:44.854724 1992 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:38:44.864269 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.868265 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.874018 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.877316 1962 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.877717 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.876804 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.892306 2145 raft_consensus.cc:2354] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:38:44.870796 2654 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.895418 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.896509 1992 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 3: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:44.867074 4364 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:38:44.902885 2680 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:44.857112 4205 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:44.911798 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:44.913570 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:44.909693 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.916640 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:44.921476 4380 raft_consensus.cc:979] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:44.922070 4439 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:44.923477 2139 raft_consensus.cc:2379] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:44.924657 2110 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:44.907445 4339 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:44.927430 2136 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:44.918025 2108 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.909943 4353 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:38:44.927439 2138 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260927 19:38:44.935451 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:38:44.941521 2145 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:44.948619 4205 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:44.953778 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 83060.7us
I20260927 19:38:44.954746 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.954900 4383 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:44.955514 4383 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:44.956175 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.934911 1989 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:44.960466 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:44.961500 2111 leader_election.cc:398] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6.
W20260927 19:38:44.961862 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 97393.9us
I20260927 19:38:44.962821 2654 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:44.922073 4380 raft_consensus.cc:1067] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
W20260927 19:38:44.978679 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 71017us
W20260927 19:38:44.982801 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 47889.9us
I20260927 19:38:44.979333 4355 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:44.963086 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.985715 2111 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.989585 4208 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:44.963307 4217 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:44.973054 4337 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:44.962620 4383 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:44.976446 4402 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:44.991019 2654 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.992947 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:44.992812 4406 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:44.970013 4439 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:44.993667 2111 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:44.995837 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.998955 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.981432 4384 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:45.000329 4217 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.007799 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.008107 4217 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.010000 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:45.010656 2111 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.013959 4341 raft_consensus.cc:2781] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:45.014534 4341 raft_consensus.cc:686] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:45.009068 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:44.994342 4355 raft_consensus.cc:686] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:45.032889 2654 leader_election.cc:398] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:38:45.033850 4337 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:45.034369 4337 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:45.034821 4337 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:45.035208 4436 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:38:45.037752 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:45.038035 4373 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:45.039757 4384 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.038386 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:45.043804 2108 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:45.015173 4164 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:45.034386 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:45.057159 2143 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:44.963804 2145 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:45.037252 4436 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:38:44.995405 3990 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.037523 4403 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:45.029811 4208 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:45.055794 4363 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:45.062553 4164 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:45.058558 4402 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:45.058983 4373 raft_consensus.cc:686] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:45.094866 4384 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:45.110698 4208 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.066659 4394 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260927 19:38:45.057947 4383 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:45.140883 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.148102 2136 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
W20260927 19:38:45.142046 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48046.4us
I20260927 19:38:45.069310 4436 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
W20260927 19:38:45.156136 1961 leader_election.cc:334] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.004s (SENT)
I20260927 19:38:45.139333 1988 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:45.130564 4164 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:45.174513 1962 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.187867 4333 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:45.188484 4333 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:45.188884 4333 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:45.199932 2138 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:45.145769 4363 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.204108 4383 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.260573 2142 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260927 19:38:45.262703 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 183923us
I20260927 19:38:45.130089 4341 consensus_queue.cc:237] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.269073 2070 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:45.245644 4384 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:45.221890 4335 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260927 19:38:45.267215 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.216248 4355 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.291291 4400 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:45.328064 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:45.327798 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.331162 2142 raft_consensus.cc:2354] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:38:45.331091 2069 raft_consensus.cc:2362] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6. Re-sending same reply.
I20260927 19:38:45.250837 4436 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.338474 2066 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 19:38:45.329190 4384 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.321931 3990 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.346617 1962 leader_election.cc:398] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:38:45.346997 1962 leader_election.cc:401] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 19:38:45.335610 2070 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:45.264559 4373 consensus_queue.cc:237] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.364930 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:45.367390 2687 raft_consensus.cc:2379] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:45.328562 4403 raft_consensus.cc:686] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:45.371349 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.362195 4456 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:45.372419 4448 log.cc:924] Time spent T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
I20260927 19:38:45.298882 4402 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.385461 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.395952 4384 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:45.397962 2687 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:45.376070 4363 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.402813 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.403451 2066 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 2.
I20260927 19:38:45.402640 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.410625 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:45.411705 2680 raft_consensus.cc:2379] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:45.411792 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.412742 2067 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:45.419338 4383 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.420289 4363 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:45.421222 4261 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.073s
I20260927 19:38:45.422638 4465 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:45.423338 4465 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:45.402704 4414 log.cc:924] Time spent T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260927 19:38:45.393790 4456 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.428439 4456 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.435633 4465 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.436105 4463 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:45.436740 4463 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:45.438437 4455 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260927 19:38:45.438992 4463 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.441767 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.442545 2066 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:38:45.443578 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.443778 4403 consensus_queue.cc:237] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.444697 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.402654 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.445724 2071 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:45.448518 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.467916 2066 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:45.457871 1988 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:45.448102 2062 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:45.472785 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.474232 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.481211 2070 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:45.470342 3990 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:38:45.445814 4350 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:45.482965 2141 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:38:45.488425 2067 raft_consensus.cc:2379] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.490592 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.492578 2066 raft_consensus.cc:2379] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.495555 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.496441 2071 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:45.500263 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.501672 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.447341 4333 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.511564 2062 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.445312 4164 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.516376 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.517228 2066 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:45.517575 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 50052.1us
I20260927 19:38:45.516017 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.521000 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.524492 2139 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:45.526458 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.523793 2067 raft_consensus.cc:2362] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4. Re-sending same reply.
W20260927 19:38:45.525259 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 132869us
W20260927 19:38:45.534085 2653 leader_election.cc:334] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.854s (SENT)
I20260927 19:38:45.539220 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.539950 2070 raft_consensus.cc:2362] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6. Re-sending same reply.
I20260927 19:38:45.547325 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.550377 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.551451 2062 raft_consensus.cc:2379] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.547992 2070 raft_consensus.cc:2362] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6. Re-sending same reply.
I20260927 19:38:45.550452 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.552338 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.552582 2067 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:45.553115 2062 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.554530 2070 raft_consensus.cc:1226] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:45.514783 4350 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:45.574266 4473 log.cc:924] Time spent T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260927 19:38:45.586337 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.582265 4456 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:45.595484 4337 consensus_meta.cc:287] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.560s user 0.012s sys 0.001s
I20260927 19:38:45.596130 4333 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.595984 4337 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.598798 4164 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:45.600251 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:45.601039 2144 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:45.610705 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:45.614878 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.605980 2071 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:45.603744 4456 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.692646 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.696400 1995 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:45.699872 2062 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:45.702180 2144 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:45.695716 2066 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:45.703423 1962 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.703382 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.705811 4464 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:45.706557 4464 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.709786 4333 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:45.711587 4456 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.711911 4333 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:45.712558 4333 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:45.714162 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.714833 2065 raft_consensus.cc:2362] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4. Re-sending same reply.
I20260927 19:38:45.715839 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.716392 2065 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:45.717450 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.718732 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.719791 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.720533 2063 raft_consensus.cc:2379] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.722882 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.723970 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.724566 2066 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:45.725730 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.726588 2062 raft_consensus.cc:2379] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.726832 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.727501 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.728219 2062 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:45.727609 2069 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:45.735064 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.710197 4337 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.721168 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.729025 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.744731 2062 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:45.749995 2071 raft_consensus.cc:2379] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.750154 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.751111 2066 raft_consensus.cc:2379] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.751195 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.751854 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.752048 2071 raft_consensus.cc:2379] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:45.743624 2063 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:45.757750 4456 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:45.758410 4456 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:45.760474 4480 log.cc:924] Time spent T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260927 19:38:45.760673 4456 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:45.765563 2071 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:45.781066 4350 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:45.769164 4436 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:38:45.787405 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:45.800254 2111 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.802770 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:45.810520 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:45.811596 2137 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:45.812507 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 109159us
I20260927 19:38:45.814092 2063 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:45.823755 2111 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.831863 2111 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:45.797772 4464 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.836172 4373 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:45.838439 4403 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:45.839154 4403 raft_consensus.cc:686] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:45.840319 4403 consensus_queue.cc:237] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.843711 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:45.844411 1988 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:45.845305 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:45.845463 3990 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:45.847337 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.848186 2138 raft_consensus.cc:2379] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:45.856124 4048 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:45.856745 4048 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.857499 4402 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:45.858269 4402 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.859270 4048 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:45.862330 4402 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:45.863031 4464 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:45.863919 4464 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:45.864964 3990 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:45.866431 3990 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:45.871059 4464 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:45.830363 3776 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:45.875321 2142 raft_consensus.cc:1261] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:45.879175 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 1162) took 1014 ms. Trace:
I20260927 19:38:45.856153 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.898723 2139 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:45.880144 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:45.889693 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.890291 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.890475 2065 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:38:45.890630 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.901676 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.896821 2069 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:45.904583 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:45.898386 2141 rpcz_store.cc:276] 0927 19:38:44.864388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:44.864855 (+ 467us) service_pool.cc:227] Handling call
0927 19:38:45.878968 (+1014113us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400142654. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:45.879103 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":290246400}
I20260927 19:38:45.911718 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.899711 2685 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:45.916673 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.921849 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:45.922426 2065 raft_consensus.cc:2362] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4. Re-sending same reply.
I20260927 19:38:45.921918 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:45.928313 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:45.932960 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:45.933504 2138 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:45.929976 2062 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:45.938823 1962 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:45.940327 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.932566 2063 raft_consensus.cc:2362] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5. Re-sending same reply.
I20260927 19:38:45.941951 1959 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:45.961406 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:45.990620 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 51873.6us
I20260927 19:38:45.980779 2069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:38:46.014359 2069 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:46.015116 2069 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:46.002216 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 65953.6us
I20260927 19:38:46.004504 2063 pending_rounds.cc:76] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 2
I20260927 19:38:46.033643 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.039919 2685 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:46.043895 4333 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.050849 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.051816 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.052996 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:45.959858 3776 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:46.055222 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.072943 2141 raft_consensus.cc:2379] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:46.055847 2138 raft_consensus.cc:2379] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:46.030478 2063 pending_rounds.cc:97] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 5.3
W20260927 19:38:45.990687 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 452624us
I20260927 19:38:46.059273 2110 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:46.059719 4475 log.cc:924] Time spent T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260927 19:38:45.997013 4456 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:46.060570 2136 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:38:46.064090 1890 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 3 to 4, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 4 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:46.067502 4356 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:46.025516 4402 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:46.076511 1995 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:46.068562 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.077486 1996 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:46.071028 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.069514 2139 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 19:38:46.074049 2063 raft_consensus.cc:2866] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:46.087347 2137 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:45.936417 4208 consensus_queue.cc:1035] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.121s
I20260927 19:38:46.086014 4402 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:46.100785 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.100133 4402 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:46.100380 4461 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260927 19:38:46.106282 4310 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260927 19:38:46.109061 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52490.2us
W20260927 19:38:46.111889 2653 leader_election.cc:334] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.023s (SENT)
I20260927 19:38:46.116791 2654 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:46.116904 2110 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:46.118305 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.119143 1995 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:46.086609 4356 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:46.091566 4338 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:46.109036 4456 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.129495 2064 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:46.148145 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.132098 1890 catalog_manager.cc:5495] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
W20260927 19:38:46.161913 4486 log.cc:924] Time spent T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.112s user 0.002s sys 0.000s
I20260927 19:38:46.165031 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 256) took 5130 ms. Trace:
I20260927 19:38:46.166003 4348 consensus_queue.cc:1046] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:46.166749 4388 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:46.167096 4348 raft_consensus.cc:3032] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:46.142333 4456 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.167516 4348 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:46.167483 4388 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.125725 4356 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:46.102335 4400 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20260927 19:38:46.174733 4400 consensus_queue.cc:1225] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer 69f09866d0a1477585f05b2f82ff4c67 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260927 19:38:46.179247 1988 raft_consensus.cc:2379] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:46.166661 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:46.191207 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.130354 4436 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:46.202057 4335 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:46.202292 1890 catalog_manager.cc:5495] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:46.206650 2685 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260927 19:38:46.212611 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 86094.9us
I20260927 19:38:46.204973 1890 catalog_manager.cc:5495] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 4 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:46.177459 4350 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260927 19:38:46.216926 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 50377us
I20260927 19:38:46.229949 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:46.229941 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:46.239461 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.241842 2136 pending_rounds.cc:76] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 5
I20260927 19:38:46.242183 2136 pending_rounds.cc:97] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260927 19:38:46.242506 2136 raft_consensus.cc:2866] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:46.240975 2069 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.168706 4348 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.254971 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:46.255919 1890 catalog_manager.cc:5495] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 3, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:46.269016 2069 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.261569 4261 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:46.214932 4436 raft_consensus.cc:686] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:46.226014 4333 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.280316 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:46.264616 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.267493 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.259480 4402 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.165526 1891 rpcz_store.cc:276] 0927 19:38:41.033976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:41.034118 (+ 142us) service_pool.cc:227] Handling call
0927 19:38:46.164796 (+5130678us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:42.829498 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:42.841621 (+ 12123us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:42.841658 (+ 37us) write_op.cc:435] Acquired schema lock
0927 19:38:42.841675 (+ 17us) tablet.cc:660] Decoding operations
0927 19:38:43.352099 (+510424us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:43.352197 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:43.352237 (+ 40us) tablet.cc:683] Acquiring locks for 31 operations
0927 19:38:43.416648 (+ 64411us) tablet.cc:699] Row locks acquired
0927 19:38:43.416671 (+ 23us) write_op.cc:260] PREPARE: finished
0927 19:38:43.416816 (+ 145us) write_op.cc:270] Start()
0927 19:38:43.416922 (+ 106us) write_op.cc:276] Timestamp: P: 1790537923416768 usec, L: 0
0927 19:38:43.416944 (+ 22us) op_driver.cc:331] REPLICATION: starting
0927 19:38:43.434171 (+ 17227us) log.cc:842] Serialized 12137 byte log entry
0927 19:38:44.552653 (+1118482us) op_driver.cc:444] REPLICATION: finished
0927 19:38:44.729929 (+177276us) write_op.cc:301] APPLY: starting
0927 19:38:44.730041 (+ 112us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:44.730703 (+ 662us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:44.730836 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:44.866205 (+135369us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:44.901651 (+ 35446us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=31
0927 19:38:44.901757 (+ 106us) write_op.cc:312] APPLY: finished
0927 19:38:44.903386 (+ 1629us) log.cc:842] Serialized 206 byte log entry
0927 19:38:44.915479 (+ 12093us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:44.928349 (+ 12870us) write_op.cc:454] Released schema lock
0927 19:38:44.945520 (+ 17171us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:44.946045 (+ 525us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:44.946091 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3242,"reported_tablets":45,"child_traces":[["op",{"apply.queue_time_us":176894,"apply.run_cpu_time_us":12529,"apply.run_wall_time_us":217120,"num_ops":31,"prepare.queue_time_us":369911,"prepare.run_cpu_time_us":57604,"prepare.run_wall_time_us":1058970,"raft.queue_time_us":231694,"raft.run_cpu_time_us":33236,"raft.run_wall_time_us":108224,"replication_time_us":1135543,"thread_start_us":755814,"threads_started":4,"wal-append.queue_time_us":540904,"wal-append.run_cpu_time_us":16156,"wal-append.run_wall_time_us":1952432}]]}
I20260927 19:38:46.274672 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.283110 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.284031 2141 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:46.286800 1962 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:46.288058 4464 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:46.288627 4464 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.291663 4325 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:46.292109 4325 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:46.292526 4325 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.294674 4402 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.278474 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet da05114b545f4ec9a25e86d9c059fea3 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:46.299292 4464 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.302703 4439 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:46.302930 4325 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.285405 4261 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.304646 4439 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:46.305001 4325 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.307364 4261 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.248025 4363 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260927 19:38:46.235576 4310 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.074s
I20260927 19:38:46.285714 4217 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:46.310573 4217 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:46.239001 3776 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.311071 4217 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:46.314932 4436 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.320966 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.321933 2138 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:38:46.308995 4348 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:46.243103 4388 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.325029 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.327284 1996 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
W20260927 19:38:46.323910 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 55408.5us
I20260927 19:38:46.329213 2654 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:46.320052 2110 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.898s (SENT) [suppressed 33 similar messages]
I20260927 19:38:46.319679 4048 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:46.335450 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:46.336059 2685 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:46.334246 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.332573 4406 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:46.335702 4048 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:46.341725 4363 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:46.346143 4406 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:46.425457 2138 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260927 19:38:46.421092 2685 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:46.332257 4356 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.368920 4048 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:46.429203 1959 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:46.422968 4484 log.cc:924] Time spent T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20260927 19:38:46.427443 4406 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:46.458527 2685 raft_consensus.cc:1226] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:46.448753 4402 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:46.448062 3990 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260927 19:38:46.446018 4363 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:46.484253 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 254806us
W20260927 19:38:46.486613 4498 log.cc:924] Time spent T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260927 19:38:46.488942 2071 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:46.489795 4468 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:46.490607 4468 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.486716 4363 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.472158 4402 raft_consensus.cc:686] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:46.494967 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.496431 2138 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 9: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:46.507468 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:46.508260 2686 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:38:46.514599 4356 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.517940 1959 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.519055 4217 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.522969 4217 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.520196 4048 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.525717 4048 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.471189 4403 consensus_queue.cc:1046] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260927 19:38:46.536839 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.540604 2138 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:46.541684 4468 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:46.544193 4505 tablet_replica.cc:330] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:38:46.545660 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:46.547487 4350 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:46.548002 4350 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.549639 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.550232 2144 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:46.550520 4350 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.552074 4208 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:46.552608 4208 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.575699 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:46.576665 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 80887.7us
I20260927 19:38:46.578414 2144 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:46.539501 3990 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:38:46.579561 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.577435 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:46.577883 4508 log.cc:924] Time spent T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.064s user 0.002s sys 0.000s
I20260927 19:38:46.585347 1962 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:46.588620 1961 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.577894 4217 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:46.587654 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:46.606034 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 135108us
I20260927 19:38:46.580526 4208 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.605770 4217 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.594638 2687 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:46.612304 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:46.600991 4406 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.581300 3990 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:46.514681 4402 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.611619 4463 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:46.614029 4310 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:46.617504 4048 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:46.611927 4388 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:46.623921 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:46.617125 4502 raft_consensus.cc:2741] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260927 19:38:46.620496 3990 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:38:46.626179 4388 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:46.632900 1993 raft_consensus.cc:1226] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:46.634274 4406 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.637562 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.638196 2137 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:46.641531 1961 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.642406 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:46.642980 1996 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:46.626767 4310 raft_consensus.cc:686] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:46.644670 4356 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260927 19:38:46.645493 2034 consensus_peers.cc:487] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db442f51d759423ba3bde8876ae46aa0. This is attempt 1: this message will repeat every 5th retry.
I20260927 19:38:46.645097 4310 consensus_queue.cc:237] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.640161 4388 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.651367 2137 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:38:46.652123 4505 raft_consensus.cc:2229] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260927 19:38:46.659627 4505 raft_consensus.cc:2258] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260927 19:38:46.652483 4217 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.653870 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:46.654697 4504 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.139s user 0.000s sys 0.002s
W20260927 19:38:46.643378 4500 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260927 19:38:46.657523 4436 consensus_queue.cc:1046] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260927 19:38:46.668313 4436 raft_consensus.cc:3032] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:46.668677 4436 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:46.669618 4436 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.686538 1996 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:38:46.694440 2654 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:46.694605 2138 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:46.695353 4406 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:38:46.695844 4406 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:46.695849 1962 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:46.696427 4339 raft_consensus.cc:3032] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:46.696758 4501 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:46.697423 4501 raft_consensus.cc:686] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:46.696871 4406 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.699739 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.700722 2034 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:46.711000 4436 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:46.696906 4339 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:46.718556 4513 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260927 19:38:46.719295 4465 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:46.720916 4464 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:46.721318 4464 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:46.721669 4464 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.703744 4217 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:46.701666 4505 ts_tablet_manager.cc:1905] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:38:46.698258 4501 consensus_queue.cc:237] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:46.742717 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 118824us
I20260927 19:38:46.748307 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:46.758180 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:46.760187 4456 raft_consensus.cc:2741] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260927 19:38:46.761534 4388 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.770100 4388 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.771000 4456 raft_consensus.cc:2781] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:46.770789 3990 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.771548 4456 raft_consensus.cc:686] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:46.775514 4464 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.777997 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:46.778894 4464 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.779206 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:46.779803 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.780268 4502 raft_consensus.cc:2741] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260927 19:38:46.781005 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:46.781538 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:46.781587 2687 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.779558 4464 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:46.786653 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:46.788352 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:46.789564 4463 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:46.790463 4502 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:46.797137 2687 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:46.799189 2034 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:46.801287 2037 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:46.772625 4456 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.807322 1989 raft_consensus.cc:1226] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:46.809331 1994 raft_consensus.cc:1261] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:38:46.804807 4463 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:46.804857 4400 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:46.810494 4514 log.cc:924] Time spent T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260927 19:38:46.823570 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:46.828382 1994 raft_consensus.cc:1261] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:46.832032 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.757694 4339 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.838999 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:46.839682 2144 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 9: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:46.841606 1996 raft_consensus.cc:1261] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:46.825120 4463 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:46.825848 4464 raft_consensus.cc:686] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:46.827723 4465 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:46.841641 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:46.834048 4502 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:46.848747 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:46.845448 4463 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:46.855173 4517 log.cc:924] Time spent T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260927 19:38:46.830189 4400 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.858008 4464 consensus_queue.cc:237] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.848585 4465 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:46.859562 4516 log.cc:924] Time spent T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260927 19:38:46.850581 4502 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:46.843350 3990 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:46.918689 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 95274.5us
I20260927 19:38:46.921304 4523 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:46.917694 4502 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.916031 4465 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.929476 4456 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:46.930123 4456 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.932417 4456 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:46.932994 4523 raft_consensus.cc:686] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:46.934006 4523 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.938189 2687 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:38:46.939446 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:46.941289 1988 raft_consensus.cc:1261] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:46.940122 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:46.939247 4363 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:46.942963 2680 raft_consensus.cc:1226] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:46.942584 4456 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:46.943205 4363 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:46.945611 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.945667 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:46.946542 4456 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:46.947882 1959 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.954262 3990 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:38:46.954761 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:46.954896 3990 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:46.956748 4339 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:46.957238 3990 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:46.961753 4363 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:46.963004 2141 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:46.982833 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:46.989943 4456 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:46.970568 4406 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260927 19:38:47.006278 4400 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:47.008544 1992 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
W20260927 19:38:47.009876 4524 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260927 19:38:47.009747 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.016425 4505 ts_tablet_manager.cc:1918] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20260927 19:38:47.017061 4505 log.cc:1196] T da05114b545f4ec9a25e86d9c059fea3 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/da05114b545f4ec9a25e86d9c059fea3
I20260927 19:38:47.017897 4502 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.019869 4502 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:47.023728 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 52644.7us
I20260927 19:38:47.026131 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.028102 4403 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260927 19:38:47.031370 4502 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.031488 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet da05114b545f4ec9a25e86d9c059fea3 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:47.037199 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:47.037563 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.032115 4338 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260927 19:38:47.055914 4465 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:47.039042 4461 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.025707 4355 consensus_queue.cc:1046] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260927 19:38:47.063247 4355 raft_consensus.cc:3032] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:47.063686 4355 raft_consensus.cc:729] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:47.064559 4355 consensus_queue.cc:260] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.038767 4468 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260927 19:38:47.066841 4463 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.069062 4463 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:47.069887 4355 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:47.070981 4523 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.071508 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:47.074918 4465 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:47.070643 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.076924 2686 raft_consensus.cc:2354] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
W20260927 19:38:47.083314 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 44998.6us
I20260927 19:38:47.085253 2034 leader_election.cc:398] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:47.085765 2034 leader_election.cc:401] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260927 19:38:47.077064 4523 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:47.093994 4468 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260927 19:38:47.097895 4468 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:47.098277 4468 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:47.099244 4468 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.085421 4463 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:47.094555 4461 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260927 19:38:47.102082 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.094410 4465 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:47.136139 2686 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:47.100514 4468 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:47.138324 4356 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:47.145249 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.148852 4523 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.149026 4464 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:47.152005 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:47.152666 1994 raft_consensus.cc:595] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.152979 1994 raft_consensus.cc:603] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:47.154789 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.158891 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.159734 4464 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.161118 4464 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.163429 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.165521 4523 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.144465 4463 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:47.171707 4463 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:47.172695 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.172888 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:47.174124 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.175642 4523 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.177325 4523 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:47.177656 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:47.178268 1995 raft_consensus.cc:595] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.179225 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.180976 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:47.182368 1995 raft_consensus.cc:595] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.182857 1995 raft_consensus.cc:603] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:47.186421 4523 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.188064 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.190635 4523 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.192628 2141 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:47.193845 2141 raft_consensus.cc:595] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.196431 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:47.196632 1961 leader_election.cc:334] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.508s (SENT)
I20260927 19:38:47.197043 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.198642 4363 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.202695 4523 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.204174 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.207521 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.209554 4463 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:47.211062 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.207186 4523 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.216722 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:47.217458 1992 raft_consensus.cc:595] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.220989 1991 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:47.221714 1991 raft_consensus.cc:595] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.222069 1991 raft_consensus.cc:603] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:47.180125 4465 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:47.231736 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:38:47.233531 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:47.234266 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:47.238245 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 235) took 5550 ms. Trace:
I20260927 19:38:47.238036 4463 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:47.239967 4543 ts_tablet_manager.cc:927] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:47.240576 1892 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 3 to 4, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:47.250465 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.247440 1892 catalog_manager.cc:5495] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 5 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 6 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:47.260905 1992 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260927 19:38:47.262776 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.262869 4436 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:47.263746 4463 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.238848 1890 rpcz_store.cc:276] 0927 19:38:41.684184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:41.684336 (+ 152us) service_pool.cc:227] Handling call
0927 19:38:47.234006 (+5549670us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:46.492743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:46.507974 (+ 15231us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:46.508040 (+ 66us) write_op.cc:435] Acquired schema lock
0927 19:38:46.508063 (+ 23us) tablet.cc:660] Decoding operations
0927 19:38:46.509104 (+ 1041us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:46.509216 (+ 112us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:46.509256 (+ 40us) tablet.cc:683] Acquiring locks for 5 operations
0927 19:38:46.510130 (+ 874us) tablet.cc:699] Row locks acquired
0927 19:38:46.510180 (+ 50us) write_op.cc:260] PREPARE: finished
0927 19:38:46.510415 (+ 235us) write_op.cc:270] Start()
0927 19:38:46.510603 (+ 188us) write_op.cc:276] Timestamp: P: 1790537926510320 usec, L: 0
0927 19:38:46.510633 (+ 30us) op_driver.cc:331] REPLICATION: starting
0927 19:38:46.511487 (+ 854us) log.cc:842] Serialized 2094 byte log entry
0927 19:38:46.634510 (+123023us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f7910a1c5fe 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb
0927 19:38:46.966079 (+331569us) op_driver.cc:444] REPLICATION: finished
0927 19:38:47.073733 (+107654us) write_op.cc:301] APPLY: starting
0927 19:38:47.073811 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:47.073969 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:47.073990 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:47.074608 (+ 618us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:47.074788 (+ 180us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0927 19:38:47.074902 (+ 114us) write_op.cc:312] APPLY: finished
0927 19:38:47.075131 (+ 229us) log.cc:842] Serialized 47 byte log entry
0927 19:38:47.083373 (+ 8242us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:47.087719 (+ 4346us) write_op.cc:454] Released schema lock
0927 19:38:47.090331 (+ 2612us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:47.127373 (+ 37042us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:47.127802 (+ 429us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19547,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":107372,"apply.run_cpu_time_us":2640,"apply.run_wall_time_us":63176,"num_ops":5,"prepare.queue_time_us":165611,"prepare.run_cpu_time_us":20498,"prepare.run_wall_time_us":280416,"raft.queue_time_us":110062,"raft.run_cpu_time_us":15114,"raft.run_wall_time_us":67042,"replication_time_us":455224,"spinlock_wait_cycles":49243008,"thread_start_us":510554,"threads_started":4,"wal-append.queue_time_us":245510}]]}
W20260927 19:38:47.290462 2110 leader_election.cc:334] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.671s (SENT)
I20260927 19:38:47.290944 2110 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.297580 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.297345 4463 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:47.299363 4463 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:47.301041 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.301523 4465 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.298260 4502 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.302414 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:47.302733 1892 catalog_manager.cc:5495] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 3 to 4, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:47.303611 4403 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.304369 1892 catalog_manager.cc:5495] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:47.309580 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.312335 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.312580 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.312863 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.313488 2069 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:47.311522 1892 catalog_manager.cc:5495] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 5, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:47.313647 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.314018 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.314916 2071 raft_consensus.cc:2354] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:38:47.316128 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.316767 2064 raft_consensus.cc:2362] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6. Re-sending same reply.
I20260927 19:38:47.317682 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.318297 2064 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:47.319144 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.320216 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.320849 2064 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 6
I20260927 19:38:47.310963 4502 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:47.321684 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.322127 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.322618 2110 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.322921 2070 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:47.323949 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.324231 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.324571 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:47.324906 2065 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:47.322546 2062 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:47.325771 4355 raft_consensus.cc:2741] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260927 19:38:47.325968 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.323949 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.325726 4468 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.327379 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.328615 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.322760 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.330478 2063 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:47.331408 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.321810 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.327400 2110 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:47.332816 2068 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:47.334270 1996 raft_consensus.cc:1226] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:47.335261 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.335640 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:47.336716 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.335723 4436 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260927 19:38:47.323037 4465 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:47.342361 2141 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:47.343889 1991 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:38:47.343849 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:47.344460 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.345111 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.346625 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:47.350934 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:47.353107 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:47.355110 2139 raft_consensus.cc:3032] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:47.372694 2139 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:47.358078 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.301254 4463 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:47.360234 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.359686 1892 catalog_manager.cc:5495] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 2, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 2 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:47.360718 2686 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:47.360721 2062 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:47.364372 1994 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:47.327409 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.335937 2068 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:38:47.378916 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.379848 2062 raft_consensus.cc:2379] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
W20260927 19:38:47.379709 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52391.6us
I20260927 19:38:47.380544 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.380678 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.378432 4502 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:47.369445 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:47.381942 1992 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:47.383121 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:47.383684 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:47.384163 2140 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:38:47.384397 2137 raft_consensus.cc:2379] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:47.384435 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.385017 4523 consensus_queue.cc:1046] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:47.385679 2144 raft_consensus.cc:2354] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:47.385545 4502 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:38:47.386025 4523 raft_consensus.cc:3032] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:47.386400 4523 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:47.355706 4333 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:47.375959 2142 raft_consensus.cc:1226] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:47.388087 2138 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260927 19:38:47.379036 2070 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:47.381244 2062 raft_consensus.cc:2362] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7. Re-sending same reply.
I20260927 19:38:47.379017 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.391034 2065 raft_consensus.cc:2362] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7. Re-sending same reply.
I20260927 19:38:47.391914 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.392935 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.393101 2068 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:47.390465 2069 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:47.396111 2037 leader_election.cc:398] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:47.396070 2067 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260927 19:38:47.396693 2037 leader_election.cc:401] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:47.394256 2141 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260927 19:38:47.397912 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.397915 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:47.399363 2136 raft_consensus.cc:2379] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:47.398550 1892 catalog_manager.cc:5495] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 5 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:47.399453 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.400497 2069 raft_consensus.cc:2362] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6. Re-sending same reply.
I20260927 19:38:47.387339 2139 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.402176 2065 raft_consensus.cc:2362] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6. Re-sending same reply.
I20260927 19:38:47.403528 2139 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:47.404467 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.405071 1992 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:47.405704 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.406206 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.410830 1996 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:47.376653 4403 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.415535 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.419888 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:47.421487 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.422439 2069 raft_consensus.cc:2354] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:47.423257 2063 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:47.433027 4261 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:47.445483 2066 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 19:38:47.448099 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 45402.8us
I20260927 19:38:47.398237 4465 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.451880 1892 catalog_manager.cc:5495] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 1 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:47.457404 4217 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:47.408030 4356 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:47.463481 2064 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:47.468122 2686 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260927 19:38:47.476992 2064 raft_consensus.cc:1261] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:47.401229 4464 consensus_queue.cc:1046] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260927 19:38:47.408766 4502 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260927 19:38:47.482070 2067 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:47.457907 4217 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.485318 2139 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:47.476055 4528 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:47.488756 2064 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:47.398406 4523 consensus_queue.cc:260] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.493425 2064 raft_consensus.cc:1261] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:38:47.497788 2064 raft_consensus.cc:1261] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:47.500828 4217 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:47.507287 2064 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:47.521746 4465 consensus_queue.cc:1046] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:47.480450 4543 tablet_copy_client.cc:316] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.130:45515
I20260927 19:38:47.535616 4456 raft_consensus.cc:3032] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:38:47.536064 4456 raft_consensus.cc:729] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:47.536856 2686 raft_consensus.cc:1226] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:38:47.485047 4310 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260927 19:38:47.476120 4356 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:47.546361 4356 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:47.553942 4463 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:47.554658 4463 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.556818 4463 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:47.560177 2070 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:47.560556 2068 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:47.561497 2062 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260927 19:38:47.563593 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 251660us
I20260927 19:38:47.568303 2067 raft_consensus.cc:1261] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260927 19:38:47.584829 1961 leader_election.cc:334] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.656s (SENT)
I20260927 19:38:47.585480 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:47.595064 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 180407us
I20260927 19:38:47.536772 4456 consensus_queue.cc:260] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:47.600962 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.602936 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:47.625440 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.630970 2139 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:47.634876 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.640668 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.642859 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.656767 2653 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:47.661957 1994 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:47.619057 4463 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.625519 4363 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:47.624667 4464 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:47.637578 4325 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.635838 4468 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.683203 2653 leader_election.cc:398] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6.
I20260927 19:38:47.616396 4502 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260927 19:38:47.692031 2071 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:47.692699 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:47.693759 2068 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:47.694957 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.696049 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.686192 4456 raft_consensus.cc:3037] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:47.716940 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.718156 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:47.718991 2062 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
W20260927 19:38:47.720971 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 99793.9us
I20260927 19:38:47.687752 4555 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:47.662374 4339 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260927 19:38:47.684144 4403 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260927 19:38:47.683710 4338 consensus_queue.cc:1035] T ac5251fc915042bab4ce94b1c17deb87 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
W20260927 19:38:47.724088 4548 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.263s user 0.004s sys 0.000s
I20260927 19:38:47.731958 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:47.716540 4554 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:47.724782 4551 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:47.720151 4552 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.710453 4355 consensus_queue.cc:1035] T 2034953084704d748cf5bd91b19f1a46 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20260927 19:38:47.749264 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:47.742580 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 72290.6us
I20260927 19:38:47.717044 4468 consensus_queue.cc:1035] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260927 19:38:47.652511 4528 consensus_queue.cc:1159] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4e0335d1a85f421380029b95a91bf0c4" responder_term: 8 status { last_received { term: 8 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 19:38:47.710824 4464 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:47.746402 4048 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:47.745879 4436 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.753062 4551 raft_consensus.cc:686] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
W20260927 19:38:47.813817 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 64665.9us
I20260927 19:38:47.728696 4555 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:47.819734 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 128837us
I20260927 19:38:47.787707 4400 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:47.823572 1961 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.835709 1961 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.810707 4464 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.813244 4556 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:47.827705 4400 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:47.757275 4554 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.853487 4333 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:47.876106 4565 log.cc:924] Time spent T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.068s user 0.001s sys 0.001s
I20260927 19:38:47.860543 4400 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:47.881255 4436 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:47.882727 4563 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:38:47.901706 4515 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.090s user 0.001s sys 0.001s
W20260927 19:38:47.913379 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 78035.2us
I20260927 19:38:47.916744 1961 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:47.921128 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 72550.3us
I20260927 19:38:47.900823 4436 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:47.930053 1961 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.935034 1961 leader_election.cc:398] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:38:47.924113 4573 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:47.934338 3990 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 19:38:47.937712 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 242811us
I20260927 19:38:47.903716 4551 consensus_queue.cc:237] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.942826 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.927143 4555 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:47.952432 2081 tablet_copy_service.cc:140] P 4cc2df048f424e08803a390946004df0: Received BeginTabletCopySession request for tablet db442f51d759423ba3bde8876ae46aa0 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:60518)
W20260927 19:38:47.954136 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 258122us
I20260927 19:38:47.954972 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.955687 1961 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.953063 2081 tablet_copy_service.cc:161] P 4cc2df048f424e08803a390946004df0: Beginning new tablet copy session on tablet db442f51d759423ba3bde8876ae46aa0 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:60518: session id = 4e0335d1a85f421380029b95a91bf0c4-db442f51d759423ba3bde8876ae46aa0
I20260927 19:38:47.960373 4523 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:47.953313 4465 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
W20260927 19:38:47.948975 4562 log.cc:924] Time spent T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20260927 19:38:47.964681 4554 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:47.966333 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:47.967505 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.967536 4464 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:47.974406 4436 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:47.974664 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:47.992718 2136 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:47.977277 2142 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260927 19:38:47.978477 2144 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:47.946008 3990 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:47.994586 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:47.998306 1961 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:48.009045 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:47.998216 3990 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:48.013021 4568 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:47.961663 4555 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20260927 19:38:48.013604 4568 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.016393 4568 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:48.021601 4502 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:38:48.023175 1962 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:48.024695 4579 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:48.025118 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:48.026681 1994 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:48.025110 4579 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:38:48.038868 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 73252.9us
I20260927 19:38:48.041472 4568 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.042081 4568 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:47.992515 4580 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:48.006739 4554 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:48.039439 4350 raft_consensus.cc:979] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:47.992853 4556 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260927 19:38:48.039724 4579 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:48.025697 4048 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.028s
I20260927 19:38:48.068784 2654 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:48.044011 4310 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260927 19:38:48.047206 4465 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:48.062117 4350 raft_consensus.cc:1067] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:48.080727 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:48.062054 4580 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:48.108243 4568 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:48.120061 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:48.123405 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 81158.2us
W20260927 19:38:48.137648 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 104618us
W20260927 19:38:48.138751 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 140583us
I20260927 19:38:48.086045 4583 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:48.139876 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:48.142788 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:48.143699 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 71388.1us
I20260927 19:38:48.117835 4580 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:48.148437 2070 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260927 19:38:48.148399 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:48.148923 2070 pending_rounds.cc:76] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 6
I20260927 19:38:48.149171 2070 pending_rounds.cc:97] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260927 19:38:48.149556 2070 raft_consensus.cc:2866] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:48.122372 4558 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.150739 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:48.151569 2064 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:48.144706 4583 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:48.145267 4048 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:48.155504 4552 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:48.151226 4589 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:48.172961 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:48.147454 4217 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
W20260927 19:38:48.174899 4584 log.cc:924] Time spent T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.141s user 0.001s sys 0.000s
I20260927 19:38:48.161861 4583 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:48.194095 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:38:48.195415 2068 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:48.195879 2068 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:48.197916 3990 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.200136 3990 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:48.185252 4356 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260927 19:38:48.202229 4363 raft_consensus.cc:979] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:48.167425 4465 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:48.217918 4591 log.cc:924] Time spent T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260927 19:38:48.156750 4558 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.219620 4400 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.220970 4558 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:48.221623 4563 raft_consensus.cc:1067] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:48.221743 4400 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:48.189632 4580 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.228349 1992 raft_consensus.cc:1226] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:48.265625 2068 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.288745 2068 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:48.282683 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 133012us
W20260927 19:38:48.302119 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 157046us
I20260927 19:38:48.301741 4563 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.310305 1994 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:48.314098 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:48.315353 1961 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:48.320622 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:48.322510 4464 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60504 (request call id 8) took 1862 ms. Trace:
I20260927 19:38:48.325166 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:48.317488 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 164957us
I20260927 19:38:48.327520 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:48.327963 2065 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:48.330816 4464 rpcz_store.cc:276] 0927 19:38:46.449972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:46.454498 (+ 4526us) service_pool.cc:227] Handling call
0927 19:38:46.506919 (+ 52421us) log.cc:842] Serialized 506 byte log entry
0927 19:38:46.653492 (+146573us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b18002d9040. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:46.715886 (+ 62394us) spinlock_profiling.cc:243] Waited 913 us on lock 0x7b18002d9040. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f79109b14ca 00007f791a092f99 00007f7910875990 00007f7910a37864 00007f7910a37155 00007f79138dc320 00007f7913942113 00007f79139687e2 00007f79188a4f39
0927 19:38:46.737672 (+ 21786us) spinlock_profiling.cc:243] Waited 8.27 ms on lock 0x7b18002d9040. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a764 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:48.298861 (+1561189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89934,"raft.queue_time_us":100933,"raft.run_cpu_time_us":1933,"raft.run_wall_time_us":9619,"spinlock_wait_cycles":27130880,"thread_start_us":44569,"threads_started":1,"wal-append.queue_time_us":110547,"wal-append.run_cpu_time_us":2464,"wal-append.run_wall_time_us":1062778}
I20260927 19:38:48.332223 2065 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:48.332722 2065 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:48.334672 4563 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.343052 4325 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:48.314338 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:48.301939 4556 consensus_queue.cc:1046] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260927 19:38:48.371137 1994 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:48.305279 4400 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260927 19:38:48.405459 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:48.335083 4310 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260927 19:38:48.381188 1989 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:48.431627 2064 raft_consensus.cc:2396] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260927 19:38:48.371739 2065 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.459152 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:48.459898 1990 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:48.461863 2065 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:48.360402 4596 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.127s
I20260927 19:38:48.466231 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:48.466382 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:48.467000 1993 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 3.
I20260927 19:38:48.467309 4563 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:48.467038 1990 raft_consensus.cc:3032] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:38:48.467988 1990 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:48.461704 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:48.469452 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 174224us
I20260927 19:38:48.381968 4609 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:48.487493 1990 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.508982 1990 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:48.509641 1994 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:48.405874 4589 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260927 19:38:48.525671 1990 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:38:48.528120 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:48.529412 4463 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:48.446274 4583 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.534198 1990 raft_consensus.cc:1261] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:48.534566 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:48.536355 4604 log.cc:924] Time spent T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
I20260927 19:38:48.520190 4465 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260927 19:38:48.544855 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:48.548372 1962 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:48.556565 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:48.540194 4463 raft_consensus.cc:686] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:48.541016 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:48.553184 4465 consensus_queue.cc:1035] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260927 19:38:48.580253 2680 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:48.571218 4523 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:48.594975 4631 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260927 19:38:48.596299 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48045us
I20260927 19:38:48.583540 4400 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:38:48.593427 4617 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260927 19:38:48.593989 4625 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:48.614005 4583 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:48.622377 4463 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:48.647697 4579 consensus_meta.cc:287] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.580s user 0.007s sys 0.004s
I20260927 19:38:48.620611 4555 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.606207 4523 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.653502 4555 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.656476 4599 raft_consensus.cc:3032] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEADER]: Stepping down as leader of term 5
W20260927 19:38:48.660146 4560 log.cc:924] Time spent T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.776s user 0.005s sys 0.000s
I20260927 19:38:48.662828 2686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34930 (request call id 449) took 1114 ms. Trace:
I20260927 19:38:48.657068 4599 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:48.675127 4599 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.676229 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:48.676644 4599 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:48.676960 2063 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 4
I20260927 19:38:48.678962 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:48.681128 1990 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:48.663326 2686 rpcz_store.cc:276] 0927 19:38:47.548529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:47.557500 (+ 8971us) service_pool.cc:227] Handling call
0927 19:38:47.557885 (+ 385us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:47.558226 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:47.558255 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:47.558678 (+ 423us) log.cc:842] Serialized 26 byte log entry
0927 19:38:47.665871 (+107193us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:47.665925 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:47.666011 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:48.659430 (+993419us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:48.662424 (+ 2994us) raft_consensus.cc:1671] finished
0927 19:38:48.662455 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:48.662727 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1767,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":21534,"spinlock_wait_cycles":14161792,"thread_start_us":106779,"threads_started":1,"wal-append.queue_time_us":284526}
I20260927 19:38:48.657280 4555 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:48.648221 4579 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.700645 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:48.697003 1995 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
W20260927 19:38:48.707394 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 133648us
I20260927 19:38:48.715389 4464 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:48.715996 4464 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:48.720337 4464 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:48.723505 4523 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:48.728538 2063 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:48.736585 4579 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:48.739476 2081 tablet_copy_source_session.cc:205] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Tablet Copy: opened 0 blocks and 1 log segments
W20260927 19:38:48.743247 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 587674us
I20260927 19:38:48.747046 2680 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
W20260927 19:38:48.748019 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 571163us
I20260927 19:38:48.748193 4578 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.748653 4578 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:48.750770 4578 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:48.751909 2138 raft_consensus.cc:1261] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:38:48.752866 2142 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:48.747583 4350 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.764670 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:48.765471 1995 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:48.761950 1989 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260927 19:38:48.763252 2653 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:48.762638 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:48.769467 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:48.770504 2145 raft_consensus.cc:3032] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:48.770821 2145 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
W20260927 19:38:48.806838 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 59745us
W20260927 19:38:48.808704 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 55858.6us
I20260927 19:38:48.820370 2070 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
W20260927 19:38:48.804828 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 107220us
I20260927 19:38:48.824405 2145 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.851955 2145 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:48.857832 1995 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:48.807775 4583 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:48.890690 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:48.892185 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 66078.7us
I20260927 19:38:48.913558 2680 raft_consensus.cc:2379] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
W20260927 19:38:48.913841 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 98855.5us
I20260927 19:38:48.921485 2145 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
W20260927 19:38:48.928293 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 92924.8us
I20260927 19:38:48.805073 4350 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:48.890581 4339 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:49.154808 4339 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:49.113484 4592 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.963s user 0.010s sys 0.000s
W20260927 19:38:49.156126 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 230009us
I20260927 19:38:49.156991 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1189) took 1194 ms. Trace:
I20260927 19:38:49.157379 2139 rpcz_store.cc:276] 0927 19:38:47.962827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:47.963320 (+ 493us) service_pool.cc:227] Handling call
0927 19:38:47.963831 (+ 511us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:47.964053 (+ 222us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:47.964068 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:47.990868 (+ 26800us) log.cc:842] Serialized 506 byte log entry
0927 19:38:48.055624 (+ 64756us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:48.055664 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:48.055818 (+ 154us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:49.108084 (+1052266us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:49.156698 (+ 48614us) raft_consensus.cc:1671] finished
0927 19:38:49.156719 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:49.156934 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12506880,"thread_start_us":63677,"threads_started":1,"wal-append.queue_time_us":126435}
I20260927 19:38:49.159031 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:49.159507 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 236176us
I20260927 19:38:48.890115 4590 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.143s
W20260927 19:38:49.142139 4602 log.cc:924] Time spent T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.984s user 0.004s sys 0.000s
I20260927 19:38:49.120682 4580 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.150082 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:48.927361 4582 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:48.925801 4456 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:49.152585 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:49.171669 2680 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
W20260927 19:38:48.972474 4637 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260927 19:38:48.942946 4583 raft_consensus.cc:686] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:49.159588 2139 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:49.145483 2136 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:49.169514 2138 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
W20260927 19:38:49.187007 4642 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.357s user 0.001s sys 0.000s
I20260927 19:38:49.187273 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:49.187944 2139 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:49.170625 4555 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:49.206486 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:49.211052 4464 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.209838 2139 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:49.217200 1962 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:49.213853 2136 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
W20260927 19:38:49.197736 4605 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260927 19:38:49.226226 1962 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
W20260927 19:38:49.236075 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 74716.2us
I20260927 19:38:49.239267 2108 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:49.236008 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 47175us
I20260927 19:38:49.196656 4644 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.195097 4339 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:49.269474 4338 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 19:38:49.269695 4635 log.cc:924] Time spent T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.341s user 0.001s sys 0.000s
I20260927 19:38:49.270295 4543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db442f51d759423ba3bde8876ae46aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:49.269004 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:49.270663 4630 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:49.188205 4456 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.224591 4555 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.170645 4580 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.281169 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1582) took 1228 ms. Trace:
I20260927 19:38:49.235146 4614 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:49.173393 4582 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.287556 2111 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:38:49.234432 4463 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:49.278625 4338 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:49.289328 2062 rpcz_store.cc:276] 0927 19:38:48.047854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:48.047985 (+ 131us) service_pool.cc:227] Handling call
0927 19:38:48.055922 (+ 7937us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:48.061411 (+ 5489us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:48.061457 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:48.070446 (+ 8989us) log.cc:842] Serialized 52 byte log entry
0927 19:38:48.137626 (+ 67180us) raft_consensus.cc:1631] Marking committed up to 7
0927 19:38:48.166667 (+ 29041us) log.cc:842] Serialized 14 byte log entry
0927 19:38:48.169380 (+ 2713us) log.cc:842] Serialized 14 byte log entry
0927 19:38:48.169680 (+ 300us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:48.169737 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:48.219089 (+ 49352us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b1800394840. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:48.596158 (+377069us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b4800271158. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79188097f1 00007f79188008e7 00007f791880077a 00007f79188f495f 00007f79188f4627 00007f7918908b2c 00007f7918908ad8 00007f7918908a58 00007f7918908a13 00007f7918907cbc 00007f7917889be6
0927 19:38:48.721997 (+125839us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b1800394840. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a638 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:49.100959 (+378962us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:49.276180 (+175221us) raft_consensus.cc:1671] finished
0927 19:38:49.276207 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:49.276370 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45613,"prepare.queue_time_us":65128,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":3282,"raft.queue_time_us":7227,"raft.run_cpu_time_us":6567,"raft.run_wall_time_us":382589,"spinlock_wait_cycles":24764800,"thread_start_us":31926,"threads_started":1,"wal-append.queue_time_us":36701}
I20260927 19:38:49.328796 2111 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:49.291827 4649 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:49.337838 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:49.341107 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:49.347000 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 64143.5us
I20260927 19:38:49.348023 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:49.331224 4338 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:49.299397 4614 raft_consensus.cc:686] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:49.357614 4571 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:49.357695 4468 consensus_queue.cc:1046] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:49.358163 4566 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:49.353785 4523 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:49.361620 4650 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:38:49.287678 4630 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:49.370675 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 144588us
I20260927 19:38:49.287293 4583 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.339619 4465 consensus_queue.cc:1035] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260927 19:38:49.342083 4590 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260927 19:38:49.325197 4463 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.364820 4571 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:49.369724 4566 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:49.286461 4644 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.374480 4339 raft_consensus.cc:3032] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:49.398743 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:49.386574 4633 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260927 19:38:49.387735 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:49.389647 4655 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:49.392616 4048 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:49.383076 4571 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:49.395798 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1594) took 1239 ms. Trace:
I20260927 19:38:49.396623 4662 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 19:38:49.397639 4652 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260927 19:38:49.385057 2680 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 19:38:49.401047 4464 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.403035 1996 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:49.416280 4647 log.cc:924] Time spent T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20260927 19:38:49.347730 4649 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.384742 4566 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:49.423264 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2284) took 1024 ms. Trace:
I20260927 19:38:49.423655 4630 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.401853 4655 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:49.424799 4655 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:49.398561 4339 raft_consensus.cc:729] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:49.408418 4582 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.409935 4614 consensus_queue.cc:237] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.411087 4636 raft_consensus.cc:979] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:49.406066 2064 rpcz_store.cc:276] 0927 19:38:48.156353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:48.164889 (+ 8536us) service_pool.cc:227] Handling call
0927 19:38:49.395574 (+1230685us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b5400393fd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:49.395712 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":591806848}
I20260927 19:38:49.413306 4048 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.417946 4662 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.419322 2144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2286) took 1018 ms. Trace:
I20260927 19:38:49.419610 4664 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260927 19:38:49.423671 2141 rpcz_store.cc:276] 0927 19:38:48.398522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:48.398910 (+ 388us) service_pool.cc:227] Handling call
0927 19:38:48.403017 (+ 4107us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:38:49.122157 (+719140us) raft_consensus.cc:1520] Early marking committed up to index 8
0927 19:38:49.161590 (+ 39433us) log.cc:842] Serialized 14 byte log entry
0927 19:38:49.326883 (+165293us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:38:49.338863 (+ 11980us) log.cc:842] Serialized 52 byte log entry
0927 19:38:49.339011 (+ 148us) raft_consensus.cc:1631] Marking committed up to 8
0927 19:38:49.339067 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:49.339151 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:49.423030 (+ 83879us) raft_consensus.cc:1671] finished
0927 19:38:49.423049 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:49.423208 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":671883,"prepare.queue_time_us":56733,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":509600,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":45263,"thread_start_us":391761,"threads_started":3,"wal-append.queue_time_us":200280}
I20260927 19:38:49.437073 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:49.466007 1995 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:49.439296 4589 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:49.441756 1989 raft_consensus.cc:1226] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:49.463176 2144 rpcz_store.cc:276] 0927 19:38:48.401052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:48.401194 (+ 142us) service_pool.cc:227] Handling call
0927 19:38:48.401525 (+ 331us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:48.817053 (+415528us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:48.817124 (+ 71us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:48.830892 (+ 13768us) log.cc:842] Serialized 26 byte log entry
0927 19:38:49.201717 (+370825us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:38:49.201752 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:49.201838 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:49.389383 (+187545us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b18002f4820. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a3a799 00007f7910a3db11 00007f7910a3dac9 00007f7910a3da59 00007f7910a3da21 00007f7910a3cd1d 00007f7919c9e204 00007f7919c9e039 00007f7910a1cdc6
0927 19:38:49.417516 (+ 28133us) raft_consensus.cc:1671] finished
0927 19:38:49.417538 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:49.419256 (+ 1718us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":415129,"raft.queue_time_us":463735,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":30862,"spinlock_wait_cycles":7665664,"thread_start_us":230269,"threads_started":1,"wal-append.queue_time_us":427083}
W20260927 19:38:49.443997 4645 log.cc:924] Time spent T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.159s user 0.000s sys 0.002s
I20260927 19:38:49.441798 4644 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.467496 2111 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:49.418366 4555 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.480630 1996 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:49.481225 1996 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60296 (request call id 1176) took 1170 ms. Trace:
I20260927 19:38:49.481525 1996 rpcz_store.cc:276] 0927 19:38:48.311196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:48.311448 (+ 252us) service_pool.cc:227] Handling call
0927 19:38:49.387574 (+1076126us) spinlock_profiling.cc:243] Waited 9.84 ms on lock 0x7b5800003398. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:49.481146 (+ 93572us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22630784}
I20260927 19:38:49.482636 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:49.483099 4048 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:49.484715 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:49.485008 4463 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:49.485409 1990 raft_consensus.cc:595] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:49.485764 4580 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:49.486828 4663 raft_consensus.cc:1067] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:49.486459 4566 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.488260 4339 consensus_queue.cc:260] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.496392 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:49.489078 4566 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.464704 4662 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:49.501065 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:49.503438 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:49.489609 4582 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:49.506769 4582 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:49.507226 4582 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:49.500519 4461 consensus_queue.cc:1159] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 19:38:49.509967 4338 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.510455 4649 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.512207 4338 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:49.514539 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:49.504771 4523 raft_consensus.cc:686] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:49.521413 4523 consensus_queue.cc:237] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.526927 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "794b9dbf053c445393588ff152553db5"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:49.535950 1990 raft_consensus.cc:595] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:38:49.535681 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 168632us
I20260927 19:38:49.549603 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:49.501847 4339 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:49.552464 2069 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 2.
I20260927 19:38:49.520008 4582 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.551791 4571 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.560899 4571 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.560562 4582 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.564589 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:49.565198 1990 raft_consensus.cc:595] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:49.565385 4671 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:49.566165 4671 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.565531 1990 raft_consensus.cc:603] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:49.572366 4350 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.572808 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:49.573443 2064 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:49.575246 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83f97251c21d4e979bce8419c208369c"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:49.576107 2145 raft_consensus.cc:595] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:49.580798 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:49.581578 1996 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:49.590128 2064 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:49.582381 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:49.597224 1992 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:49.593874 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:49.594951 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:49.594478 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:49.595490 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 224328us
I20260927 19:38:49.609210 2065 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
W20260927 19:38:49.610029 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 127477us
I20260927 19:38:49.611908 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:49.612600 2145 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:38:49.624202 1992 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:49.583338 4655 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.626446 1990 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 19:38:49.628089 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:49.727499 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:49.728152 2144 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:49.648499 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:49.648864 1962 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:49.691296 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 57038.3us
I20260927 19:38:49.637677 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:49.681658 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:49.733268 2141 raft_consensus.cc:2354] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:49.737555 2070 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:49.738059 2070 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:49.609427 4664 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:49.744221 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 95458.9us
I20260927 19:38:49.745976 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:49.746156 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:49.745098 2138 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:38:49.750840 2682 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:49.750770 1996 raft_consensus.cc:595] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:49.753259 2139 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 19:38:49.760442 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:49.707837 4575 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:49.707609 4669 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.761260 1996 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:49.694262 4261 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.726517 4671 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.615357 4589 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.770228 2070 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.607045 4662 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.748250 4563 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:49.774067 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:49.779876 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:49.799831 1988 raft_consensus.cc:2354] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:49.801018 2111 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:49.815588 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:49.774840 4261 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:49.842893 4468 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:49.847257 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:49.847937 4672 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.848619 4672 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.851056 4672 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:49.852350 4680 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260927 19:38:49.854882 4631 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.855386 4631 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.771957 4575 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.845571 1961 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:49.861892 4575 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.826319 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:49.826436 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:49.865402 1994 raft_consensus.cc:595] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:49.830195 4463 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.831308 4555 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.832106 4644 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.832940 4590 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.833719 4658 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.841130 4552 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260927 19:38:49.870359 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 44145.3us
I20260927 19:38:49.878146 2682 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
W20260927 19:38:49.872102 4634 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260927 19:38:49.760936 4664 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.865614 4468 raft_consensus.cc:686] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:49.859241 4625 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:49.886443 1961 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:49.900103 2108 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:49.799239 4589 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.773666 4669 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.907500 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:49.961704 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 119356us
I20260927 19:38:49.878557 4463 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.968307 2037 leader_election.cc:398] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:49.994531 2037 leader_election.cc:401] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:49.995457 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:49.967046 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 67023.9us
I20260927 19:38:49.898298 4333 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:38:49.999723 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:49.898937 4590 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.005985 4664 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.006071 4590 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.006836 4456 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.008500 4463 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.998886 4672 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 5
I20260927 19:38:49.896852 4555 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:50.010443 4678 log.cc:924] Time spent T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.163s user 0.003s sys 0.000s
I20260927 19:38:50.011456 4555 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.002605 2069 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:50.012552 4636 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:49.970258 4575 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:49.902918 4631 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:49.970156 4577 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.015024 4577 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.013775 4636 raft_consensus.cc:979] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:50.017022 4636 raft_consensus.cc:1067] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:50.018251 4575 raft_consensus.cc:686] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:50.019385 4589 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.019259 4575 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.023011 4577 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.023217 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:49.969982 4675 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
W20260927 19:38:49.969326 4677 log.cc:924] Time spent T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260927 19:38:49.889057 4644 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:49.963645 4655 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:49.997633 4614 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.018334 4561 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 19:38:50.046154 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.001473 4523 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.030009 4664 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260927 19:38:50.060350 4601 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:49.904290 4658 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.063324 4658 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.063915 4621 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.064476 4621 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.060421 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.066540 4621 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:50.067572 4679 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260927 19:38:50.068004 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.094836 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:50.095742 2682 raft_consensus.cc:2354] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:50.110252 2034 leader_election.cc:398] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:50.110756 2034 leader_election.cc:401] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260927 19:38:50.049713 4561 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:50.113847 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:50.107584 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 221258us
I20260927 19:38:49.999933 4625 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:50.119422 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 171852us
I20260927 19:38:50.097765 4662 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.114741 4672 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:50.118400 4261 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.040104 4468 consensus_queue.cc:237] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.129577 2065 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
W20260927 19:38:50.131278 4339 consensus_meta.cc:287] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.569s user 0.009s sys 0.000s
I20260927 19:38:50.135329 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:38:50.136459 4569 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:50.137787 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:50.139075 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:50.139848 2066 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:50.141659 4669 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.144366 4691 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.144727 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.148351 4621 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:50.148939 4621 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.152432 2070 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.158491 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:50.159287 2071 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:50.183077 4644 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.061012 4601 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.136963 4569 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.185052 4601 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:50.186789 4569 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:50.187486 4685 log.cc:924] Time spent T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260927 19:38:50.054203 4614 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.186733 4625 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.189349 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.189785 4614 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.190937 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.199546 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.200160 2139 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:50.115737 4561 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 2 FOLLOWER]: Advancing to term 3
W20260927 19:38:50.191606 4543 consensus_meta.cc:287] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.746s user 0.012s sys 0.002s
I20260927 19:38:50.210513 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.093601 4523 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.217124 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.144868 4691 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.224330 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:38:50.233610 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 77592.3us
W20260927 19:38:50.262472 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 165201us
I20260927 19:38:50.450431 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.311233 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.303226 1987 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:50.451362 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:50.326203 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.310313 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.313372 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.287401 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.462760 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.492794 2110 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:50.344101 2065 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260927 19:38:50.530447 2139 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260927 19:38:50.530992 2139 pending_rounds.cc:76] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Aborting all ops after (but not including) 3
I20260927 19:38:50.531284 2139 pending_rounds.cc:97] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260927 19:38:50.531605 2139 raft_consensus.cc:2866] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:50.528291 2071 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:50.533631 2062 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
W20260927 19:38:50.535593 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 85454.6us
I20260927 19:38:50.541196 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:50.555228 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 62552.4us
W20260927 19:38:50.437938 4670 log.cc:924] Time spent T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.746s user 0.004s sys 0.000s
I20260927 19:38:50.286563 4662 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:50.553268 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.564920 2140 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:50.564003 2110 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:50.341877 4678 log.cc:924] Time spent T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260927 19:38:50.568122 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.569960 1990 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:50.571044 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.572643 2037 leader_election.cc:398] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:50.335888 4355 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:50.313686 4628 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.307018 4573 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:50.534641 4664 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5.
I20260927 19:38:50.486407 4551 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:50.318347 4461 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:50.591612 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.506561 4669 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:50.505931 4543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:34930 (request call id 444) took 3343 ms. Trace:
I20260927 19:38:50.546061 4456 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.573690 4594 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260927 19:38:50.613559 4456 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.613315 4543 rpcz_store.cc:276] 0927 19:38:47.163087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:47.163282 (+ 195us) service_pool.cc:227] Handling call
0927 19:38:47.240311 (+ 77029us) ts_tablet_manager.cc:928] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
0927 19:38:48.803449 (+1563138us) tablet_copy_client.cc:361] Tablet copy session begun
0927 19:38:49.444544 (+641095us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:49.444863 (+ 319us) tablet_copy_client.cc:490] Wrote new tablet metadata
0927 19:38:50.232901 (+788038us) tablet_copy_client.cc:411] Wrote new consensus metadata
0927 19:38:50.376212 (+143311us) tablet_replica.cc:205] Creating consensus instance
0927 19:38:50.477291 (+101079us) ts_tablet_manager.cc:983] Replica registered.
0927 19:38:50.505841 (+ 28550us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":164983,"tablet-copy.queue_time_us":76007,"thread_start_us":139382,"threads_started":2}
I20260927 19:38:50.615626 4456 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.587175 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.578568 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.585228 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.616855 1987 raft_consensus.cc:2379] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:50.588481 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.617707 2653 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.618475 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.589039 2062 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:50.620281 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.486840 4465 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:50.564356 4578 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:50.594053 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.535651 4325 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:50.604773 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.606899 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.606989 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.595520 2066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "794b9dbf053c445393588ff152553db5"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:50.611815 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.574186 4589 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:50.575428 4655 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:50.612777 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.688967 2066 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:50.621824 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.623418 2064 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260927 19:38:50.691857 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 53582us
I20260927 19:38:50.623449 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.571810 4693 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:50.689992 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:50.693423 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 81666.6us
I20260927 19:38:50.694239 2066 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:50.546182 4561 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.697007 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.699182 1961 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.573781 4662 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.615428 4543 tablet_copy_client.cc:791] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:50.710602 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.624704 4523 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.718081 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:50.686847 4569 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:50.590831 4628 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.585081 4355 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.738873 2686 raft_consensus.cc:1226] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 2. Current term is 3. Ops: []
I20260927 19:38:50.597492 4590 consensus_queue.cc:1046] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.126s
I20260927 19:38:50.689940 4363 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260927 19:38:50.754468 4595 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:50.739543 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.764585 4628 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.765225 4561 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:50.717864 4048 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.048s
I20260927 19:38:50.766551 4662 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.695990 4464 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:50.697276 4658 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:50.743673 4625 raft_consensus.cc:979] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:50.744233 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:50.700443 4638 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.051s sys 0.135s
I20260927 19:38:50.721203 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.700497 4631 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:50.714553 4626 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:50.591250 4573 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.702651 4601 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:50.747079 4543 tablet_copy_client.cc:655] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:50.749418 1996 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:50.752244 4355 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.767326 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.610301 4551 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.620779 4461 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.720022 1992 raft_consensus.cc:3032] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:50.696535 4325 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.775133 4621 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.779973 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.767498 2142 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:50.802320 1993 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:50.625350 4669 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.787973 2066 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.705613 4656 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260927 19:38:50.868676 1992 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:50.872772 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 293082us
I20260927 19:38:50.877311 1992 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.878921 1992 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:50.784425 4638 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.051s sys 0.139s
I20260927 19:38:50.888547 4551 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.890136 1992 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
W20260927 19:38:50.890301 4697 log.cc:924] Time spent T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.184s user 0.000s sys 0.001s
I20260927 19:38:50.891607 4628 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:38:50.891770 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 170665us
I20260927 19:38:50.892495 1961 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.893654 4628 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:50.893972 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.895977 2066 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:50.872944 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 167273us
W20260927 19:38:50.872835 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 174458us
I20260927 19:38:50.703099 4465 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:50.875347 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 102967us
I20260927 19:38:50.875784 2064 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:50.895321 2065 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
W20260927 19:38:50.905576 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 324844us
I20260927 19:38:50.906805 2110 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:50.908702 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:50.909750 4619 raft_consensus.cc:1067] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:50.910956 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.910862 4563 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:50.913726 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:50.907505 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:50.914769 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.917632 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.915223 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:50.877277 4048 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:50.803764 4464 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.925256 2110 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:50.927590 1989 raft_consensus.cc:2379] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:50.927753 2654 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:50.918035 4655 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:50.918817 4619 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:50.940644 4465 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.941653 4564 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:50.942083 4564 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:50.929662 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.942412 4564 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:50.864179 4631 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.801129 4658 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.950716 4644 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:50.951344 4644 raft_consensus.cc:686] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:50.951830 4573 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.953102 4671 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:50.924365 4688 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:50.953843 4688 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.954819 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.956069 4669 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.956416 4688 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.957476 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:50.952286 4644 consensus_queue.cc:237] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.958034 2070 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:50.958761 2070 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:50.959921 1962 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.961512 4577 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:50.961648 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.962733 4461 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:50.963405 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:50.964300 4464 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.964766 2654 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:50.954371 4325 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:50.969700 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:50.972795 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 49564.5us
I20260927 19:38:50.974818 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:50.965612 4567 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260927 19:38:50.979907 2070 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:50.983599 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.984735 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:50.985984 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.988154 1992 raft_consensus.cc:2379] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:50.990513 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:50.991351 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.923342 4406 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260927 19:38:50.995831 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.994195 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:50.997267 1992 raft_consensus.cc:3032] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Stepping down as leader of term 2
I20260927 19:38:50.997676 1992 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:50.992345 2070 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.993068 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:50.930663 4048 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:50.923820 4551 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260927 19:38:51.030066 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.031432 2110 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:51.033676 4650 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:51.037189 4048 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.038329 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:51.038595 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.040009 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:50.994879 4625 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.039681 2110 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:51.042659 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:51.043696 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.043839 4575 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:51.044390 4575 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.044725 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.044833 4575 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:50.977303 4581 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:51.045908 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.047232 2143 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:51.014202 4310 raft_consensus.cc:979] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:51.050129 4310 raft_consensus.cc:1067] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:51.014626 4658 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:50.992841 4631 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.055812 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.056713 2066 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
W20260927 19:38:51.060258 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 439947us
I20260927 19:38:51.062438 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.016933 4564 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.019130 4543 tablet_copy_client.cc:531] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:51.020139 1959 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:51.026276 4616 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.073287 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:51.040294 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.040403 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.060333 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.063952 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.060647 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.065418 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.066448 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:51.069046 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 48986.1us
I20260927 19:38:51.076223 2065 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:51.069690 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.076830 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:51.076960 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.077411 2145 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:51.074169 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.078192 2144 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:51.078691 2142 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:38:51.079222 2142 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:51.079278 1962 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:51.079610 2142 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:51.078137 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:50.998313 1992 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.083648 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.083841 1961 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:51.083803 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.085794 1961 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.086236 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.089610 2680 raft_consensus.cc:2354] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7.
I20260927 19:38:51.070520 4658 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:51.172825 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:51.173972 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 90268.4us
I20260927 19:38:51.178010 2145 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:38:51.119638 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:51.181579 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.109288 1992 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 2 FOLLOWER]: Advancing to term 3
I20260927 19:38:51.087788 4048 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:51.099181 4596 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:51.186908 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.189631 2686 raft_consensus.cc:2379] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 3: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:51.189970 2037 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.121325 2142 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.195439 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.200799 1992 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:51.200524 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.202868 2142 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:51.086953 4669 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.184788 4523 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:51.085958 4464 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:51.186709 4644 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.206483 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.118041 4595 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:51.207939 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:51.256071 1988 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:51.208295 4468 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:51.198922 4552 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:51.185938 4590 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:51.189204 4585 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.204842 4589 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260927 19:38:51.201217 4664 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.196084 4625 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:51.184624 4658 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.228111 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:51.216692 4543 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:51.228899 1989 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:51.228921 4657 raft_consensus.cc:3032] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:51.326601 4657 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:51.327564 4657 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.328595 4657 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:51.299198 4700 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.187319 4596 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:51.329552 4596 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:51.329276 4700 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.331984 4700 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:51.105422 4662 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.333628 4662 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.233220 4575 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.100760 4217 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:51.237047 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.237592 2071 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:51.220582 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.090078 4564 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:51.245007 2066 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 19:38:51.250409 2108 consensus_peers.cc:487] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51dd0aa486b149cb9fc4a9dd54e3cdf4. This is attempt 1: this message will repeat every 5th retry.
W20260927 19:38:51.346197 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 95898.8us
I20260927 19:38:51.253072 2069 raft_consensus.cc:1261] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:51.351565 2145 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260927 19:38:51.204315 4668 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.291436 4468 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.357944 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:51.358637 2686 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:51.359453 4468 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:51.244925 4688 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:51.360572 4688 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.362607 4688 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.245162 4689 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:51.363235 4689 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.247215 4572 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:51.260712 4644 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.254168 4601 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:51.365176 4689 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.251537 4464 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:51.309630 1988 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:38:51.314652 2137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1206) took 1640 ms. Trace:
I20260927 19:38:51.376292 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.314757 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:51.379398 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 64719.5us
I20260927 19:38:51.380127 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.314563 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:51.291433 4595 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.383317 4668 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 19:38:51.374313 4572 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:51.383821 4668 raft_consensus.cc:686] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:51.384678 4595 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.303956 4363 consensus_queue.cc:1046] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260927 19:38:51.337071 2141 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:51.342317 4662 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.344651 4575 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:51.356344 2034 leader_election.cc:398] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7.
I20260927 19:38:51.387151 2034 leader_election.cc:401] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:51.368744 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.388641 2145 raft_consensus.cc:2379] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:51.388893 4465 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 7
I20260927 19:38:51.370128 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.391181 2136 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:51.384706 4668 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.395025 4658 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:51.396467 4709 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.439s user 0.004s sys 0.001s
I20260927 19:38:51.371904 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.373672 2682 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:51.421605 4655 raft_consensus.cc:979] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:51.421192 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.422808 4596 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.424842 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:51.377874 4601 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.429000 4601 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:51.400183 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.375417 2137 rpcz_store.cc:276] 0927 19:38:49.674226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:49.731367 (+ 57141us) service_pool.cc:227] Handling call
0927 19:38:49.759582 (+ 28215us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:49.860341 (+100759us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:49.860384 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:50.023688 (+163304us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b2c00063708. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:38:50.024091 (+ 403us) log.cc:842] Serialized 26 byte log entry
0927 19:38:50.249862 (+225771us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:50.581598 (+331736us) log.cc:842] Serialized 14 byte log entry
0927 19:38:50.873439 (+291841us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:51.314257 (+440818us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400051754. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:51.314330 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:51.314389 (+ 59us) raft_consensus.cc:1671] finished
0927 19:38:51.314408 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:51.314580 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4708,"prepare.queue_time_us":440475,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":15447,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":14030,"spinlock_wait_cycles":240084096,"thread_start_us":798653,"threads_started":3,"wal-append.queue_time_us":581753}
I20260927 19:38:51.401995 4572 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:51.403923 4658 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:51.433245 4658 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:51.433712 4658 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:51.405645 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.406679 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:51.407230 1962 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:51.407279 4595 consensus_queue.cc:1046] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:51.408638 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:51.439838 1996 raft_consensus.cc:3032] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:51.440266 1996 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:51.443216 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.382033 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:51.415470 4691 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.444670 4662 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.445653 4436 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:51.446156 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:51.447423 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 65445us
I20260927 19:38:51.448364 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:51.449815 4700 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:51.450080 4468 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:51.450759 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:51.382472 1988 raft_consensus.cc:1261] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:51.398847 4644 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:51.443866 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.414443 2145 raft_consensus.cc:1226] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:51.422147 4655 raft_consensus.cc:1067] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:51.460261 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.460281 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.462496 2145 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:38:51.378659 4464 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:51.464762 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:51.465086 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:51.466595 1992 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:51.469408 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:51.469475 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.470101 2136 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:51.469522 1993 raft_consensus.cc:3032] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:51.471203 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.471050 1891 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:51.470918 1993 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:51.470639 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:51.371325 2686 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:51.474172 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:51.475497 2108 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:51.476995 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.472625 1891 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:51.485033 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.490000 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.493896 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.425226 4596 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.446158 4436 raft_consensus.cc:686] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:51.513616 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.457134 4625 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:51.527469 4456 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:51.526551 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.533114 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.491235 4701 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:51.539314 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 47821.8us
I20260927 19:38:51.512463 4589 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.519735 1993 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.547078 1993 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:51.549537 4350 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
W20260927 19:38:51.552363 4563 consensus_queue.cc:1159] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "69f09866d0a1477585f05b2f82ff4c67" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260927 19:38:51.544814 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 69422.3us
I20260927 19:38:51.539765 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.475078 4338 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260927 19:38:51.530102 4585 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.530788 4688 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:51.566829 2136 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:51.541030 4523 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:51.542222 4569 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.567173 1962 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.574568 4350 raft_consensus.cc:686] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:51.539383 4552 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.572587 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:51.577034 2140 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:51.542985 4596 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:51.536616 4625 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:51.571630 4725 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.053s user 0.001s sys 0.001s
I20260927 19:38:51.594942 2067 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:51.520572 4702 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260927 19:38:51.609215 1962 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:51.584173 4669 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.512902 4721 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260927 19:38:51.613945 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:51.592938 4689 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:51.593001 4465 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7.
I20260927 19:38:51.617985 4523 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:51.626338 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:51.592238 4581 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260927 19:38:51.628661 4350 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.634101 4217 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:38:51.615140 4668 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.620487 4596 raft_consensus.cc:2781] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:51.639801 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:51.641539 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.642225 2071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83f97251c21d4e979bce8419c208369c"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:38:51.668219 2071 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:51.668641 2071 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:51.592509 4569 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.652539 2062 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:38:51.653529 4465 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:51.653728 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.654436 4601 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.607842 4436 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:51.645547 4657 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:51.639582 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:51.661245 4523 raft_consensus.cc:686] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:51.694267 4657 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:51.694815 4657 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:51.694700 4523 consensus_queue.cc:237] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.697552 4601 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:51.703591 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 118005us
I20260927 19:38:51.673441 1993 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:51.704082 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.706743 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:51.702697 2071 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.711309 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:51.712389 2071 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.718681 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:51.719733 1987 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:51.720222 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:51.720827 2145 raft_consensus.cc:3032] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:51.721104 2145 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:51.723899 2145 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.725322 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:51.686782 4723 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260927 19:38:51.709983 1996 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.737982 1996 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:51.728255 2110 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:51.733678 4601 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.637015 4650 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260927 19:38:51.734890 4461 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:51.741899 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:51.684355 4569 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:51.744375 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:51.721669 4552 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.721602 4662 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:51.756735 1996 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:38:51.684613 4465 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.742669 4596 raft_consensus.cc:686] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:51.726333 4589 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260927 19:38:51.759413 4721 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:51.843835 4596 consensus_queue.cc:237] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:51.763607 4609 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260927 19:38:51.864782 1987 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:51.871744 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
W20260927 19:38:51.835273 4627 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.007s sys 0.002s
I20260927 19:38:51.835148 4722 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:38:51.910065 1987 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260927 19:38:51.872809 4657 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:51.885414 4627 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.922s user 0.007s sys 0.002s
W20260927 19:38:51.939906 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 71556.1us
I20260927 19:38:51.920627 4668 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:51.948148 2066 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:51.877576 4658 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:51.930250 4551 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260927 19:38:51.924842 4464 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.072571 2065 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260927 19:38:51.967756 4668 raft_consensus.cc:686] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:52.121932 2069 raft_consensus.cc:3037] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:52.136003 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:52.085345 4596 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:52.144343 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:52.146514 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.150810 4664 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:52.160070 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.033s sys 0.074s
W20260927 19:38:52.160543 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.033s sys 0.074s
W20260927 19:38:52.160876 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 572809us
I20260927 19:38:52.187719 2069 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:52.126233 4738 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:52.192934 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 145724us
I20260927 19:38:52.145327 4465 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.197973 4738 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:52.205919 4579 raft_consensus.cc:979] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:52.148900 4596 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.220155 2064 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:52.206018 4657 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.221586 4609 raft_consensus.cc:1067] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:52.237396 4658 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:52.267549 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 82038.8us
I20260927 19:38:52.269192 1994 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260927 19:38:52.265738 4700 consensus_meta.cc:287] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.816s user 0.012s sys 0.000s
W20260927 19:38:52.275679 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 129257us
I20260927 19:38:52.278779 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:52.279431 2140 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:52.280855 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:52.286845 2140 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:38:52.287318 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:52.288105 2139 raft_consensus.cc:2354] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:52.288458 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:52.266161 4464 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.290516 2037 leader_election.cc:398] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:52.292272 2037 leader_election.cc:401] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 19:38:52.213305 4668 consensus_queue.cc:237] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.303154 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:52.304416 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:52.318852 4738 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:52.330127 2139 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:52.276830 4579 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:38:52.332145 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.332168 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:52.333376 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:52.289820 4700 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:52.347533 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60518 (request call id 352) took 1033 ms. Trace:
I20260927 19:38:52.347760 4589 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:52.347960 2062 rpcz_store.cc:276] 0927 19:38:51.314161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:51.314325 (+ 164us) service_pool.cc:227] Handling call
0927 19:38:52.347378 (+1033053us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54000fd554. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:52.347469 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":565289856}
I20260927 19:38:52.334153 4579 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:52.334436 4625 consensus_meta.cc:287] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.721s user 0.011s sys 0.000s
I20260927 19:38:52.350032 4625 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:52.335718 1961 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.336817 2063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:52.352084 2063 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:52.352435 2063 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:52.294075 4583 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260927 19:38:52.357308 4579 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.358474 4579 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:52.340900 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:52.333199 4688 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:52.361253 4688 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:52.361814 1987 raft_consensus.cc:1226] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:52.341442 2140 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:52.296869 4657 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
W20260927 19:38:52.344092 4739 log.cc:924] Time spent T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20260927 19:38:52.296132 4590 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:52.364238 4590 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:52.365227 4590 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.367808 4596 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.368211 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:52.381047 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 59443.4us
I20260927 19:38:52.389465 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:52.391791 2685 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:38:52.387176 4639 consensus_queue.cc:1046] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260927 19:38:52.394464 4325 consensus_queue.cc:1046] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:52.392248 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:52.390851 2063 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:52.397805 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 62197.6us
I20260927 19:38:52.398216 1992 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:52.399895 2140 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:52.405848 2063 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.406811 2110 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:52.407964 2069 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 19:38:52.408002 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:52.417536 4581 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:52.418213 4581 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:52.418772 4575 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:52.417543 4650 consensus_queue.cc:1046] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260927 19:38:52.420644 4650 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:52.421083 4650 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:52.418932 4644 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:52.424362 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:52.452479 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 53793.4us
I20260927 19:38:52.454775 2067 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:52.423066 4363 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:52.462677 2654 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.463052 2071 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:52.463694 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:52.483939 2139 raft_consensus.cc:2379] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:52.487610 1994 raft_consensus.cc:1226] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:52.421928 4650 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:38:52.498720 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 45382.2us
W20260927 19:38:52.502228 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 95209.1us
I20260927 19:38:52.490164 4556 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260927 19:38:52.504189 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 116592us
I20260927 19:38:52.464572 4363 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:52.461813 4585 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260927 19:38:52.460975 4707 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
W20260927 19:38:52.511333 4750 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20260927 19:38:52.526167 2139 raft_consensus.cc:1261] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:52.529706 4688 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.530843 4583 raft_consensus.cc:3032] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:52.535197 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:52.554901 2063 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:52.535723 2685 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260927 19:38:52.568053 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 100626us
I20260927 19:38:52.569036 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:52.572080 2654 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.542091 4363 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:52.573344 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.575819 1996 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:38:52.584551 4751 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:52.561924 4624 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:52.586575 4624 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.588707 4624 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.594450 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:52.553544 4217 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:52.605250 4217 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.605870 4746 consensus_queue.cc:1046] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:52.562150 4583 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:52.566923 4590 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:52.600472 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:52.601732 4563 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:52.602905 4763 ts_tablet_manager.cc:927] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.612099 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:52.607935 4217 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.580958 4666 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:52.582635 4261 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:38:52.586670 4688 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:52.696265 4583 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:52.731029 4583 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:52.623881 4590 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.736178 4590 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:52.804922 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 235776us
I20260927 19:38:52.814064 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.764950 4650 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:52.864714 1891 catalog_manager.cc:5495] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 6, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 6 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:52.862605 1961 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.867376 1891 catalog_manager.cc:5495] T c6886530f9f741c2acaff96949f65f19 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:52.876582 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:52.877478 2139 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:52.843778 4590 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
W20260927 19:38:52.876921 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44036.9us
W20260927 19:38:52.882839 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 315331us
I20260927 19:38:52.888721 1991 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:38:52.843442 4551 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:52.870489 4564 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:52.957530 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:52.973229 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:52.989188 2065 raft_consensus.cc:1261] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:53.019654 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:52.897953 4333 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:53.041740 1961 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:53.042414 2136 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:52.958189 4590 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:53.086054 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.985533 4461 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:53.087960 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:52.961722 4551 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:52.958349 4578 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
W20260927 19:38:53.125757 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 78914.4us
I20260927 19:38:53.131968 4744 consensus_queue.cc:1046] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260927 19:38:53.132898 4766 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.242s user 0.001s sys 0.000s
I20260927 19:38:53.133205 4744 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:53.077008 4763 tablet_copy_client.cc:316] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.204.131:45045
I20260927 19:38:53.078824 4217 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:53.013087 4564 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:53.111800 4333 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:38:53.145568 4543 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260927 19:38:53.120389 4523 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:53.182518 4625 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:53.184612 4543 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 19:38:53.185722 4543 tablet_bootstrap.cc:492] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:38:53.185534 4590 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.186472 4543 ts_tablet_manager.cc:1397] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 2.009s user 0.046s sys 0.033s
I20260927 19:38:53.188733 4523 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.191320 4523 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:53.111296 4699 consensus_queue.cc:1035] T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
W20260927 19:38:53.196048 4769 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
W20260927 19:38:53.197969 4770 log.cc:924] Time spent T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260927 19:38:53.113152 4461 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:53.218938 1993 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1238) took 1260 ms. Trace:
I20260927 19:38:53.231096 1993 rpcz_store.cc:276] 0927 19:38:51.952990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:51.953394 (+ 404us) service_pool.cc:227] Handling call
0927 19:38:51.975616 (+ 22222us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:52.009630 (+ 34014us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:52.009683 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:52.373210 (+363527us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b2c00028668. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:38:52.373587 (+ 377us) log.cc:842] Serialized 26 byte log entry
0927 19:38:52.611849 (+238262us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:52.611884 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:52.611946 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:53.209645 (+597699us) raft_consensus.cc:1671] finished
0927 19:38:53.209679 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:53.213678 (+ 3999us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":37545,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":10441,"spinlock_wait_cycles":121694592,"thread_start_us":237984,"threads_started":1,"wal-append.queue_time_us":459793}
I20260927 19:38:53.222829 4543 raft_consensus.cc:348] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:38:53.240027 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 193246us
I20260927 19:38:53.241927 2137 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
W20260927 19:38:53.245321 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 202402us
I20260927 19:38:53.263501 4787 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:53.264117 4787 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:53.257788 4333 consensus_queue.cc:1046] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260927 19:38:53.240408 4543 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:38:53.278350 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:53.286087 4523 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:53.286557 4523 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:53.286906 4523 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:53.296455 4523 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.304246 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:53.305239 4566 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:53.305835 4566 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.307202 4523 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:53.308866 4566 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:53.310686 2145 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:53.313861 1891 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:53.316609 1891 catalog_manager.cc:5495] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:53.336122 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2076) took 1616 ms. Trace:
I20260927 19:38:53.358364 2064 rpcz_store.cc:276] 0927 19:38:51.719545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:51.728363 (+ 8818us) service_pool.cc:227] Handling call
0927 19:38:51.728656 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:52.220814 (+492158us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:53.335732 (+1114918us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400322b54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:53.336023 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1129778944}
I20260927 19:38:53.361852 1891 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:53.413834 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:53.415454 2064 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
W20260927 19:38:53.415228 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 445722us
W20260927 19:38:53.353912 4363 consensus_meta.cc:287] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.756s user 0.003s sys 0.009s
I20260927 19:38:53.424948 2136 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260927 19:38:53.425674 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:53.439626 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 467634us
I20260927 19:38:53.483459 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:53.475980 1891 catalog_manager.cc:5495] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:53.495553 1891 catalog_manager.cc:5495] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:53.505024 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:53.432981 4363 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.625955 2145 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:53.469210 4551 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:53.499460 4668 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:53.566715 4564 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:53.627030 2137 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:53.503944 4664 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.138s
I20260927 19:38:53.677371 1891 catalog_manager.cc:5495] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 7, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
W20260927 19:38:53.728581 2037 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.131:45045 timed out after 2.013s (SENT) [suppressed 3 similar messages]
W20260927 19:38:53.680953 2145 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 1235) took 1927 ms (client timeout 2013 ms). Trace:
W20260927 19:38:53.682531 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 55492.2us
W20260927 19:38:53.637705 4779 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.494s user 0.006s sys 0.003s
I20260927 19:38:53.764542 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:53.820596 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 90223us
W20260927 19:38:53.825907 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 377382us
W20260927 19:38:53.761550 2145 rpcz_store.cc:269] 0927 19:38:51.719863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:51.719990 (+ 127us) service_pool.cc:227] Handling call
0927 19:38:53.309894 (+1589904us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b5800834eb0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138ddf63 00007f7913941aba 00007f79139593af 00007f791395ab4e 00007f7913966228 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c
0927 19:38:53.646911 (+337017us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1116856576}
I20260927 19:38:53.641808 4787 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:53.819170 4356 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:53.819695 4578 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:53.894600 1892 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 241) took 12372 ms (client timeout 15000 ms). Trace:
I20260927 19:38:53.887293 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet da05114b545f4ec9a25e86d9c059fea3 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:40144
I20260927 19:38:53.892056 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:53.903409 2139 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:53.923753 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:53.891018 4771 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:53.819259 4668 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.598520 4543 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:53.945052 4789 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:53.969070 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.004s sys 0.010s
W20260927 19:38:53.970326 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 463544us
W20260927 19:38:53.980818 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.995s user 0.005s sys 0.010s
I20260927 19:38:53.978588 2144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1247) took 1151 ms. Trace:
I20260927 19:38:53.982439 2144 rpcz_store.cc:276] 0927 19:38:52.827444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:52.827594 (+ 150us) service_pool.cc:227] Handling call
0927 19:38:52.835193 (+ 7599us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:52.844831 (+ 9638us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:52.844871 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:52.845497 (+ 626us) log.cc:842] Serialized 26 byte log entry
0927 19:38:52.852642 (+ 7145us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:52.852719 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:52.852854 (+ 135us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:53.978345 (+1125491us) raft_consensus.cc:1671] finished
0927 19:38:53.978371 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:53.978521 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9110,"raft.queue_time_us":10912,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":6561,"threads_started":1,"wal-append.queue_time_us":184819}
I20260927 19:38:53.912941 4578 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.917366 4590 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:53.945567 4786 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:53.955222 4616 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.017851 4616 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:53.987288 2071 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2077) took 2258 ms. Trace:
I20260927 19:38:53.957685 4669 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:53.990381 4363 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:53.906455 4356 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:53.980417 4787 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:53.902216 1892 rpcz_store.cc:269] 0927 19:38:41.513951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:41.514348 (+ 397us) service_pool.cc:227] Handling call
0927 19:38:53.886515 (+12372167us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:47.586966 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:47.587681 (+ 715us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:47.587723 (+ 42us) write_op.cc:435] Acquired schema lock
0927 19:38:47.587748 (+ 25us) tablet.cc:660] Decoding operations
0927 19:38:47.588965 (+ 1217us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:47.589063 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:47.589104 (+ 41us) tablet.cc:683] Acquiring locks for 8 operations
0927 19:38:47.714354 (+125250us) tablet.cc:699] Row locks acquired
0927 19:38:47.716683 (+ 2329us) write_op.cc:260] PREPARE: finished
0927 19:38:47.726839 (+ 10156us) write_op.cc:270] Start()
0927 19:38:47.732898 (+ 6059us) write_op.cc:276] Timestamp: P: 1790537927725063 usec, L: 0
0927 19:38:47.734418 (+ 1520us) op_driver.cc:331] REPLICATION: starting
0927 19:38:47.810651 (+ 76233us) log.cc:842] Serialized 3238 byte log entry
0927 19:38:48.085307 (+274656us) op_driver.cc:444] REPLICATION: finished
0927 19:38:48.343873 (+258566us) write_op.cc:301] APPLY: starting
0927 19:38:48.350646 (+ 6773us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:48.350954 (+ 308us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:48.351091 (+ 137us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:48.352133 (+ 1042us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:48.352417 (+ 284us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0927 19:38:48.352480 (+ 63us) write_op.cc:312] APPLY: finished
0927 19:38:48.352821 (+ 341us) log.cc:842] Serialized 65 byte log entry
0927 19:38:48.353024 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:48.353196 (+ 172us) write_op.cc:454] Released schema lock
0927 19:38:48.353258 (+ 62us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:48.360369 (+ 7111us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:48.362060 (+ 1691us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":221175,"reported_tablets":43,"child_traces":[["op",{"apply.queue_time_us":196037,"apply.run_cpu_time_us":3805,"apply.run_wall_time_us":59610,"num_ops":8,"prepare.queue_time_us":130120,"prepare.run_cpu_time_us":6107,"prepare.run_wall_time_us":224031,"replication_time_us":341769,"thread_start_us":284746,"threads_started":2}]]}
I20260927 19:38:53.968971 4461 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.023499 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1662) took 1166 ms. Trace:
I20260927 19:38:54.021550 2071 rpcz_store.cc:276] 0927 19:38:51.728598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:51.728766 (+ 168us) service_pool.cc:227] Handling call
0927 19:38:52.349063 (+620297us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:52.465404 (+116341us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:53.305287 (+839883us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400322154. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:53.986994 (+681707us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b5400322150. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:53.987186 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1687612928}
I20260927 19:38:54.075374 2062 rpcz_store.cc:276] 0927 19:38:52.870965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:52.871235 (+ 270us) service_pool.cc:227] Handling call
0927 19:38:52.876890 (+ 5655us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:53.152338 (+275448us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:53.152368 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:53.152422 (+ 54us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:53.152454 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:54.023222 (+870768us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b5400323f54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:54.023281 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:54.023434 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":633122432}
W20260927 19:38:54.039057 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.536s user 0.056s sys 0.102s
W20260927 19:38:54.078460 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 590031us
W20260927 19:38:53.955955 2066 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.010s sys 0.031s
W20260927 19:38:54.080396 2066 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.010s sys 0.031s
I20260927 19:38:54.065728 2139 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:53.978036 4797 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.021139 4616 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:54.145716 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 65901.7us
I20260927 19:38:54.073925 4701 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.075495 4589 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.074210 4551 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.137467 4465 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.148869 4619 raft_consensus.cc:3032] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:38:53.974059 4771 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.011246 4789 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.137545 4722 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.145649 4795 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:54.140309 4662 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.207528 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:38:54.208832 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 59919.1us
I20260927 19:38:54.076042 4639 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.147867 4738 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.168183 4754 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:54.073872 4669 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.160349 4571 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.172650 4745 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.138545 4787 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.205525 4701 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.216665 4619 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:54.286051 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 65737.9us
I20260927 19:38:54.218683 4461 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.245023 4662 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.239447 4552 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.239269 4805 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.205991 4589 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.239651 4310 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.154357 4797 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.249315 4791 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.208863 4551 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.258162 4621 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.289147 4339 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:54.300910 4794 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.244642 4639 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.215497 4465 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.022415 4590 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.295610 4363 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.024514 4786 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.244553 4738 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:54.317899 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 589332us
I20260927 19:38:54.305559 4745 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.322216 4582 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.363801 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:54.364750 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:54.321903 4552 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.323853 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2096) took 1020 ms. Trace:
I20260927 19:38:54.240976 4722 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.330176 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:54.384279 2136 raft_consensus.cc:2379] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:54.344249 2137 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:54.344843 4621 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.247802 4754 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.243064 4795 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.353536 4807 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.354135 4802 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:54.345466 4731 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 1.663s user 0.004s sys 0.015s
I20260927 19:38:54.310374 4461 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.350601 4356 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.318221 4571 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.374310 2066 rpcz_store.cc:276] 0927 19:38:53.303718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:53.354134 (+ 50416us) service_pool.cc:227] Handling call
0927 19:38:53.354574 (+ 440us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:53.355007 (+ 433us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:53.355045 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:53.355542 (+ 497us) log.cc:842] Serialized 26 byte log entry
0927 19:38:54.080947 (+725405us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:38:54.081003 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:54.081115 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:54.323599 (+242484us) raft_consensus.cc:1671] finished
0927 19:38:54.323624 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:54.323772 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11148,"raft.queue_time_us":15850,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5082,"thread_start_us":725045,"threads_started":1,"wal-append.queue_time_us":966144}
I20260927 19:38:54.350520 4563 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:54.361858 4747 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.403151 2071 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:54.396477 4752 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.341471 4310 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.408525 2140 raft_consensus.cc:2379] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:54.356496 4363 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.371622 4338 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.386493 4789 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:54.415000 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 245900us
I20260927 19:38:54.347285 4339 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.396441 4668 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.447299 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:54.451960 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:54.457144 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:54.372576 4582 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.348722 4794 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.350323 4619 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.409216 4810 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.412855 4771 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.347574 4791 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:54.451264 4803 log.cc:924] Time spent T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.203s user 0.001s sys 0.001s
I20260927 19:38:54.418442 4356 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.333931 4805 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:54.483750 4650 consensus_meta.cc:287] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.648s user 0.010s sys 0.005s
W20260927 19:38:54.489169 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 119536us
I20260927 19:38:54.463380 4825 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.469683 4789 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.496117 4701 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.522595 4771 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.490363 4787 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.489058 4809 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.483919 4621 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.552735 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:54.496487 4619 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:54.488634 4668 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:54.532357 2131 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.026s sys 0.007s
W20260927 19:38:54.597177 2131 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.689s user 0.027s sys 0.007s
I20260927 19:38:54.567292 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:54.568612 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:54.582737 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:54.462805 4338 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.444659 4563 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:54.562585 4816 log.cc:924] Time spent T 81be6adca85345a3b402f5d474ec3e5a P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.236s user 0.001s sys 0.000s
I20260927 19:38:54.611681 4701 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:54.612056 4461 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.612203 4701 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.495771 4669 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.600909 2070 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:54.600042 4551 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.607023 4339 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.609570 4829 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.610045 4571 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.610625 4795 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.495875 4792 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.411346 4802 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.595839 4702 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:54.591109 2155 tablet_copy_service.cc:140] P 69f09866d0a1477585f05b2f82ff4c67: Received BeginTabletCopySession request for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 from peer 4e0335d1a85f421380029b95a91bf0c4 ({username='slave'} at 127.0.0.1:46930)
I20260927 19:38:54.444252 4752 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.556864 4356 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:54.592650 4798 log.cc:924] Time spent T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
I20260927 19:38:54.445297 4747 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.651762 4543 ts_tablet_manager.cc:1428] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 1.465s user 0.017s sys 0.012s
I20260927 19:38:54.425102 4807 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.617189 4808 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.654186 4836 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.591625 4589 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.611682 4338 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.649605 4786 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.646222 4465 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.642503 4621 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.654755 4702 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:54.659534 4363 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.603057 4650 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.588527 4789 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.600198 4790 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.646461 4639 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.646919 4551 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.561937 4825 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.648419 4833 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.672487 4824 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.672775 4582 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.614329 4668 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.682327 4836 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.718770 4551 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.680325 4823 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.586573 4771 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.489646 4810 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.663791 4792 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.731833 4802 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:54.732879 4792 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.657783 4738 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.657558 4745 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.660804 4722 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.694953 4831 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.694726 4621 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.696520 4807 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.596297 4809 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.682611 4754 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.700431 4701 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.693745 4791 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.726090 4668 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.734925 4771 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.735016 4747 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.650337 4829 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.740358 4821 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.736509 4824 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:54.744375 4805 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.716445 4790 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.749655 4810 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.747217 4794 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.756398 4836 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.756551 4824 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.700359 4808 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.726716 4823 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.760699 4828 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:54.761526 4828 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.757171 4821 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.748085 4831 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.767364 4829 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.769260 4825 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.723891 4833 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.809777 1987 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:38:54.810484 1991 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:54.810935 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60286 (request call id 1741) took 1383 ms. Trace:
I20260927 19:38:54.816941 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:54.814898 1995 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260927 19:38:54.817116 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.821571 1991 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:38:54.812222 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.817245 1992 rpcz_store.cc:276] 0927 19:38:53.427630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:53.428087 (+ 457us) service_pool.cc:227] Handling call
0927 19:38:53.428492 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:53.428890 (+ 398us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:53.428917 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:53.429491 (+ 574us) log.cc:842] Serialized 26 byte log entry
0927 19:38:53.890469 (+460978us) raft_consensus.cc:1631] Marking committed up to 7
0927 19:38:54.299674 (+409205us) log.cc:842] Serialized 14 byte log entry
0927 19:38:54.299871 (+ 197us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:54.299939 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:54.810231 (+510292us) raft_consensus.cc:1671] finished
0927 19:38:54.810340 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:54.810724 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":99480,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":43199,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":9375,"thread_start_us":814998,"threads_started":2,"wal-append.queue_time_us":717841}
I20260927 19:38:54.814661 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.827963 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:54.829252 2069 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:38:54.826977 1987 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:54.831717 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:54.839655 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.840698 1993 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:54.843986 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.845707 1993 raft_consensus.cc:2396] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260927 19:38:54.849411 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.852699 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.685570 2688 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:54.695591 4310 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.876195 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:54.672904 4356 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:54.665458 4543 ts_tablet_manager.cc:1460] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Tablet startup took 1478ms
I20260927 19:38:54.658212 4552 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.709487 4577 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:54.712141 4650 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.881145 4577 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.885138 4808 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.879249 1994 raft_consensus.cc:2379] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:54.891657 4577 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.893949 4791 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:38:54.886883 4650 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:54.896386 4791 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:54.891114 4831 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.906256 4790 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.902878 1992 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:54.926705 4823 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.869017 4563 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.932840 4619 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:54.934199 4619 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:54.937731 4791 raft_consensus.cc:686] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:54.940061 4791 consensus_queue.cc:237] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.946264 4650 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:54.969693 4650 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:54.972239 4789 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:54.947635 4833 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:54.997660 4809 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.002264 1987 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:55.016055 4809 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:55.017800 1993 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:55.019738 4809 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:55.023485 4551 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.026901 4621 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.030741 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:54.880031 4543 ts_tablet_manager.cc:1462] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Trace:
0927 19:38:47.163087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:47.163282 (+ 195us) service_pool.cc:227] Handling call
0927 19:38:47.240311 (+ 77029us) ts_tablet_manager.cc:928] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
0927 19:38:48.803449 (+1563138us) tablet_copy_client.cc:361] Tablet copy session begun
0927 19:38:49.444544 (+641095us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:49.444863 (+ 319us) tablet_copy_client.cc:490] Wrote new tablet metadata
0927 19:38:50.232901 (+788038us) tablet_copy_client.cc:411] Wrote new consensus metadata
0927 19:38:50.376212 (+143311us) tablet_replica.cc:205] Creating consensus instance
0927 19:38:50.477291 (+101079us) ts_tablet_manager.cc:983] Replica registered.
0927 19:38:50.505841 (+ 28550us) inbound_call.cc:173] Queueing success response
0927 19:38:51.177741 (+671900us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0927 19:38:53.186808 (+2009067us) ts_tablet_manager.cc:1429] Starting tablet replica
0927 19:38:53.219984 (+ 33176us) tablet_replica.cc:259] Starting instrumentation
0927 19:38:53.220730 (+ 746us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":164983,"raft.queue_time_us":176220,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":46753,"tablet-copy.queue_time_us":76007,"tablet-download-pool.queue_time_us":6533,"tablet-download-pool.run_cpu_time_us":1111,"tablet-download-pool.run_wall_time_us":143945,"thread_start_us":489092,"threads_started":3}
I20260927 19:38:55.052695 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.053788 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:55.054221 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.058025 4619 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.068573 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.052968 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.074717 4752 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:55.098690 4828 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.102203 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:55.108925 4809 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.108925 1992 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:38:55.066128 4650 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:55.115159 4821 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.129309 4578 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:55.131445 4793 log.cc:924] Time spent T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 1.172s user 0.009s sys 0.006s
I20260927 19:38:55.142239 4590 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.143074 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.145586 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:55.146435 1996 raft_consensus.cc:2362] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5. Re-sending same reply.
I20260927 19:38:55.160472 2068 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 379) took 2205 ms. Trace:
I20260927 19:38:55.196107 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:55.197300 1987 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 3.
I20260927 19:38:55.203105 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.154827 2155 tablet_copy_service.cc:161] P 69f09866d0a1477585f05b2f82ff4c67: Beginning new tablet copy session on tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 from peer 4e0335d1a85f421380029b95a91bf0c4 at {username='slave'} at 127.0.0.1:46930: session id = 4e0335d1a85f421380029b95a91bf0c4-51dd0aa486b149cb9fc4a9dd54e3cdf4
I20260927 19:38:55.214100 4363 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:55.215155 4363 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.216614 1959 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:38:55.218663 4614 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:55.219957 4614 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.222015 4691 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:55.173051 4662 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.196686 2068 rpcz_store.cc:276] 0927 19:38:52.955110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:52.955351 (+ 241us) service_pool.cc:227] Handling call
0927 19:38:52.955864 (+ 513us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:53.151828 (+195964us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:53.151865 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:53.152339 (+ 474us) log.cc:842] Serialized 26 byte log entry
0927 19:38:53.450811 (+298472us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:53.450864 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:53.450950 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:55.129492 (+1678542us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:38:55.159683 (+ 30191us) raft_consensus.cc:1671] finished
0927 19:38:55.159712 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:55.160108 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39293,"raft.queue_time_us":189173,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":4617,"spinlock_wait_cycles":13884672,"thread_start_us":463964,"threads_started":2,"wal-append.queue_time_us":749353}
W20260927 19:38:55.231820 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 67565.7us
W20260927 19:38:55.259603 2034 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.038s sys 0.035s
W20260927 19:38:55.260185 2034 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.038s sys 0.035s
W20260927 19:38:55.260604 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 687295us
I20260927 19:38:55.260717 4702 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:55.222960 4691 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:55.285596 4797 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.286217 4828 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:55.286995 4828 raft_consensus.cc:686] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:55.255180 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:55.316854 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 80701.1us
I20260927 19:38:55.318068 2653 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:55.318322 4363 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:55.321990 1962 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.017s sys 0.068s
W20260927 19:38:55.314297 1960 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.033s sys 0.076s
I20260927 19:38:55.322100 4583 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:38:55.327662 1960 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s user 0.033s sys 0.076s
W20260927 19:38:55.328619 1962 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.785s user 0.018s sys 0.068s
W20260927 19:38:55.328979 2111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.016s sys 0.063s
W20260927 19:38:55.336361 2111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.017s sys 0.063s
I20260927 19:38:55.350349 4691 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.351339 4217 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:38:55.352360 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 136150us
I20260927 19:38:55.353418 4614 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.354689 4828 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.357827 2155 tablet_copy_source_session.cc:205] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 19:38:55.359169 4625 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:55.359157 2110 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.029s sys 0.066s
I20260927 19:38:55.360848 4400 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:55.348316 1962 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:55.360981 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:55.370788 4217 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.376443 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:55.376015 2110 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.971s user 0.029s sys 0.066s
W20260927 19:38:55.378902 2108 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s user 0.003s sys 0.043s
W20260927 19:38:55.388120 2108 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.795s user 0.004s sys 0.043s
W20260927 19:38:55.393651 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 800872us
I20260927 19:38:55.371483 4625 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:55.403076 2682 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.017s sys 0.025s
W20260927 19:38:55.403676 2682 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s user 0.018s sys 0.026s
I20260927 19:38:55.403522 2653 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:55.404935 4567 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:55.405314 4567 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:55.405723 4567 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:55.407928 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.408727 2141 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:55.408987 2144 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:55.411418 2142 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:38:55.412254 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:55.396178 2036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.012s sys 0.026s
W20260927 19:38:55.413630 2036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.012s sys 0.026s
I20260927 19:38:55.414084 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.415634 2155 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:46930 (request call id 487) took 1043 ms. Trace:
W20260927 19:38:55.416597 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 68546.6us
I20260927 19:38:55.417634 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.412950 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.416237 2155 rpcz_store.cc:276] 0927 19:38:54.391864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:54.471480 (+ 79616us) service_pool.cc:227] Handling call
0927 19:38:55.154250 (+682770us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f791887394d 00007f791797ca54 00007f791797c9e2 00007f791797c911
0927 19:38:55.399487 (+245237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8168192}
I20260927 19:38:55.419750 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:55.394869 4614 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:55.424134 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:55.425071 1990 raft_consensus.cc:595] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:55.419847 1961 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.007s sys 0.026s
W20260927 19:38:55.427908 1961 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.929s user 0.008s sys 0.027s
I20260927 19:38:55.441617 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.442898 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.396610 4400 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:55.446753 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.449889 4461 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:38:55.427944 2654 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.011s sys 0.030s
I20260927 19:38:55.450959 4792 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260927 19:38:55.465649 2654 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.980s user 0.013s sys 0.031s
I20260927 19:38:55.451350 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.452314 4845 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:55.453913 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.454908 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.456653 4625 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.457768 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet da05114b545f4ec9a25e86d9c059fea3 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:38:55.445876 2136 raft_consensus.cc:2362] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5. Re-sending same reply.
I20260927 19:38:55.468633 2138 raft_consensus.cc:2396] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260927 19:38:55.458931 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.461480 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:55.441859 4813 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.006s sys 0.025s
I20260927 19:38:55.466301 2141 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:38:55.467952 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.471169 2142 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
W20260927 19:38:55.470533 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 985199us
I20260927 19:38:55.465960 4792 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:55.481315 4813 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s user 0.007s sys 0.026s
I20260927 19:38:55.494315 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.492697 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.498040 4400 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:55.489672 1991 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:55.486614 2137 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:38:55.484069 2686 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:55.491953 4792 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:55.503132 1991 raft_consensus.cc:595] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:55.507287 1959 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:55.509043 4456 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.002s sys 0.020s
W20260927 19:38:55.509630 4456 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s user 0.003s sys 0.020s
I20260927 19:38:55.514312 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.515663 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.518086 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.519109 2684 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:55.520375 4461 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:55.522773 4461 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:55.522936 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.523288 4461 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:55.523886 4461 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.524267 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.525858 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.527340 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.528218 2142 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:55.518332 4625 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:55.529498 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.536693 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.534840 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.543038 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.545882 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.546202 2142 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:55.548362 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.545998 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.545241 2136 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:55.547776 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:55.552119 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.533650 4625 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:55.538661 4702 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:55.567785 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.538106 4217 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.533218 4845 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.598199 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:55.553853 4563 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:55.616741 4844 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:56.194000 4563 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.194389 4844 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:55.762300 4791 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.023s sys 0.048s
W20260927 19:38:55.687189 4850 log.cc:924] Time spent T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.237s user 0.000s sys 0.003s
W20260927 19:38:55.806314 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 117526us
W20260927 19:38:55.688023 2037 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.032s sys 0.049s
W20260927 19:38:56.197640 4791 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.163s user 0.036s sys 0.067s
W20260927 19:38:56.206373 2037 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.102s user 0.034s sys 0.085s
I20260927 19:38:56.136613 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:56.211819 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 1.13962e+06us
I20260927 19:38:55.553298 4456 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:55.799782 1994 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.029s sys 0.054s
W20260927 19:38:56.223582 1994 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s user 0.042s sys 0.069s
I20260927 19:38:55.563038 4625 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:56.211156 4729 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:56.012813 4567 consensus_meta.cc:287] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.607s user 0.014s sys 0.000s
W20260927 19:38:56.102416 4852 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20260927 19:38:56.196758 4563 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:55.598591 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.229413 2034 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:56.240721 4729 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:55.616307 4788 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:55.669687 4461 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:55.819077 4552 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.047s sys 0.071s
I20260927 19:38:55.657325 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.197327 4844 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:56.263926 4552 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.355s user 0.054s sys 0.089s
I20260927 19:38:55.614069 4702 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:55.819051 4325 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:56.280359 2139 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.971s user 0.029s sys 0.058s
W20260927 19:38:56.287003 2139 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.338s user 0.030s sys 0.059s
I20260927 19:38:56.283694 4844 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:56.264856 4563 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:56.277148 4788 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.250536 4567 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:56.293378 4792 consensus_meta.cc:287] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.751s user 0.011s sys 0.002s
I20260927 19:38:56.269380 4461 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:56.312134 4563 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
W20260927 19:38:56.323909 2682 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.919s user 0.030s sys 0.023s
W20260927 19:38:56.327008 2682 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.922s user 0.031s sys 0.023s
I20260927 19:38:56.329627 2685 raft_consensus.cc:2354] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:56.304108 4552 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260927 19:38:56.321190 4828 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.010s sys 0.056s
I20260927 19:38:56.307621 4325 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.306164 4456 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:56.366001 4828 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s user 0.010s sys 0.057s
I20260927 19:38:56.327394 4461 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.314471 4792 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.381525 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.371187 4563 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.388128 2686 raft_consensus.cc:2379] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:56.375667 4791 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:56.381098 4729 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:56.377601 4845 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:56.392021 4563 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:56.405707 4563 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:56.408416 2145 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.018s sys 0.037s
I20260927 19:38:56.365314 4844 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:56.410591 2145 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.955s user 0.018s sys 0.038s
I20260927 19:38:56.399219 4702 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:56.407006 4828 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:56.419960 1988 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.020s sys 0.059s
W20260927 19:38:56.420516 1988 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s user 0.020s sys 0.059s
W20260927 19:38:56.425285 4848 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.161s user 0.001s sys 0.002s
I20260927 19:38:56.395020 4791 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:56.429251 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 460) took 1004 ms. Trace:
I20260927 19:38:56.425690 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 19:38:56.454228 4456 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.471179 4325 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.423229 4792 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:56.435743 1988 rpcz_store.cc:276] 0927 19:38:55.425210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:55.425447 (+ 237us) service_pool.cc:227] Handling call
0927 19:38:55.465171 (+ 39724us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:55.493766 (+ 28595us) raft_consensus.cc:1520] Early marking committed up to index 7
0927 19:38:55.515906 (+ 22140us) log.cc:842] Serialized 14 byte log entry
0927 19:38:56.425083 (+909177us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:56.425163 (+ 80us) raft_consensus.cc:1631] Marking committed up to 7
0927 19:38:56.428842 (+ 3679us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:56.428964 (+ 122us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:56.429166 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31554,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":904621,"threads_started":1}
I20260927 19:38:56.449258 4791 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:56.510043 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 317841us
I20260927 19:38:56.448195 4828 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.512835 4792 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:56.500360 4567 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:56.518090 4456 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:56.525605 4563 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.534457 2685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34930 (request call id 480) took 1047 ms. Trace:
I20260927 19:38:56.536780 2685 rpcz_store.cc:276] 0927 19:38:55.486763 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:55.486947 (+ 184us) service_pool.cc:227] Handling call
0927 19:38:56.534246 (+1047299us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54006cc5d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:56.534400 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1903202048}
I20260927 19:38:56.514930 4792 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.531319 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.536914 4461 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:56.555249 4828 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.560230 4828 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:56.571111 4844 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:56.573022 4855 log.cc:924] Time spent T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.105s user 0.005s sys 0.001s
I20260927 19:38:56.560982 4456 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.573834 4828 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.569890 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.578109 4456 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.573212 4563 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:56.587234 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.595613 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.596606 2137 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
W20260927 19:38:56.606359 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 401467us
I20260927 19:38:56.617444 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.605414 2683 raft_consensus.cc:1226] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:56.625593 1988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
W20260927 19:38:56.625540 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 383976us
I20260927 19:38:56.624776 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1275) took 1544 ms. Trace:
W20260927 19:38:56.644601 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 44490.9us
I20260927 19:38:56.641110 1988 raft_consensus.cc:595] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:56.590374 4563 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.593289 4625 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.648265 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:56.628398 4844 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:38:56.657404 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 233255us
I20260927 19:38:56.664765 2686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34930 (request call id 482) took 1151 ms. Trace:
I20260927 19:38:56.665194 2686 rpcz_store.cc:276] 0927 19:38:55.513731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:55.534995 (+ 21264us) service_pool.cc:227] Handling call
0927 19:38:56.136391 (+601396us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5800005498. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188a41b0 00007f79188a3a94 00007f7911f8bc04 00007f7911f8bb92
0927 19:38:56.664561 (+528170us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54006027d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:38:56.664688 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":421089408}
I20260927 19:38:56.651216 4792 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:56.661091 4854 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260927 19:38:56.642351 1994 rpcz_store.cc:276] 0927 19:38:55.080359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:55.080548 (+ 189us) service_pool.cc:227] Handling call
0927 19:38:55.130524 (+ 49976us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:55.131457 (+ 933us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 19:38:55.131830 (+ 373us) log.cc:842] Serialized 14 byte log entry
0927 19:38:56.624380 (+1492550us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:56.624455 (+ 75us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:56.624515 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:56.624604 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:56.624722 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8015,"prepare.queue_time_us":793,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":165893,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":5517,"thread_start_us":1467512,"threads_started":2,"wal-append.queue_time_us":1274049}
I20260927 19:38:56.670663 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:56.672636 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:56.675623 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:56.680693 1988 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:56.682255 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.682806 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:56.683696 4804 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:56.683941 4865 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:56.684332 4865 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:56.684713 4865 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:56.685513 4792 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:56.686167 4792 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:56.688457 4859 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260927 19:38:56.688418 4792 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.689946 2145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1258) took 1286 ms. Trace:
I20260927 19:38:56.690296 2145 rpcz_store.cc:276] 0927 19:38:55.403959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:55.423223 (+ 19264us) service_pool.cc:227] Handling call
0927 19:38:55.453441 (+ 30218us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:55.454015 (+ 574us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:55.454045 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:55.454747 (+ 702us) log.cc:842] Serialized 26 byte log entry
0927 19:38:56.417984 (+963237us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:38:56.418025 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:56.418111 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:56.689718 (+271607us) raft_consensus.cc:1671] finished
0927 19:38:56.689742 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:56.689869 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":157,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"thread_start_us":962813,"threads_started":1,"wal-append.queue_time_us":1064228}
I20260927 19:38:56.692862 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.694026 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.694712 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.676256 4456 consensus_queue.cc:1046] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260927 19:38:56.663395 4844 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.703044 4844 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:56.706283 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:56.710208 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 43881.2us
I20260927 19:38:56.711256 4871 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260927 19:38:56.715101 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.715821 2679 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:56.717576 1988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "794b9dbf053c445393588ff152553db5"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:56.717818 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.718194 1988 raft_consensus.cc:595] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:56.718587 1988 raft_consensus.cc:603] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:56.719215 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:56.695681 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.698077 4828 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.720227 1990 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:38:56.730230 2144 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2426) took 1167 ms. Trace:
I20260927 19:38:56.730600 2144 rpcz_store.cc:276] 0927 19:38:55.554740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:56.560450 (+1005710us) service_pool.cc:227] Handling call
0927 19:38:56.712266 (+151816us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:56.722037 (+ 9771us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:56.722064 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:56.722105 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:56.722136 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:56.722189 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:56.722302 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260927 19:38:56.786885 4625 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:56.733575 4828 consensus_queue.cc:1046] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260927 19:38:56.883826 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:56.884532 1994 raft_consensus.cc:595] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:56.884596 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.884971 1994 raft_consensus.cc:603] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:56.887553 4800 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:56.887837 2682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34930 (request call id 478) took 2424 ms. Trace:
I20260927 19:38:56.888258 2682 rpcz_store.cc:276] 0927 19:38:54.463835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:54.463995 (+ 160us) service_pool.cc:227] Handling call
0927 19:38:54.464446 (+ 451us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:38:54.467890 (+ 3444us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:38:54.467936 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:38:54.488419 (+ 20483us) log.cc:842] Serialized 26 byte log entry
0927 19:38:55.404253 (+915834us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:38:56.328343 (+924090us) log.cc:842] Serialized 14 byte log entry
0927 19:38:56.887232 (+558889us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540055e9a0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138f5fb3 00007f7913962f0f 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c
0927 19:38:56.887284 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:56.887447 (+ 163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:38:56.887497 (+ 50us) raft_consensus.cc:1671] finished
0927 19:38:56.887595 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:56.887748 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2934,"prepare.queue_time_us":1101345,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":8494,"raft.run_cpu_time_us":5464,"raft.run_wall_time_us":42000,"spinlock_wait_cycles":323676800,"thread_start_us":1737375,"threads_started":2,"wal-append.queue_time_us":1715030}
I20260927 19:38:56.891010 4788 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:56.891793 4563 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:56.892907 4861 log.cc:924] Time spent T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.337s user 0.001s sys 0.001s
W20260927 19:38:56.894347 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 243323us
I20260927 19:38:56.893318 4625 raft_consensus.cc:3032] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:56.899744 1962 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:56.905212 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:56.908464 2139 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2400) took 2581 ms. Trace:
I20260927 19:38:56.908951 2139 rpcz_store.cc:276] 0927 19:38:54.327743 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:54.327890 (+ 147us) service_pool.cc:227] Handling call
0927 19:38:54.328209 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:38:54.948619 (+620410us) raft_consensus.cc:1520] Early marking committed up to index 4
0927 19:38:56.290369 (+1341750us) log.cc:842] Serialized 14 byte log entry
0927 19:38:56.651739 (+361370us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:38:56.651799 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:38:56.651846 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:38:56.897027 (+245181us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54002bbbd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:56.897077 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:38:56.908356 (+ 11279us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1454672,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":56,"raft.queue_time_us":759,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":1443,"spinlock_wait_cycles":1244748160,"thread_start_us":1679642,"threads_started":2,"wal-append.queue_time_us":411374}
I20260927 19:38:56.912757 4864 raft_consensus.cc:979] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:56.913489 4862 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:56.914333 4862 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.901554 4625 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:56.918414 4625 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.920102 4625 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:56.901257 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.927749 2679 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
W20260927 19:38:56.933874 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 200789us
I20260927 19:38:56.934780 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:56.906479 2110 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:56.882050 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:56.948261 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 48620.5us
I20260927 19:38:56.956549 2139 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:56.957298 2139 raft_consensus.cc:595] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:56.958501 2139 raft_consensus.cc:603] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 19:38:56.962765 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 308130us
W20260927 19:38:56.957240 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 50913.6us
I20260927 19:38:56.951768 4563 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:56.950567 4625 raft_consensus.cc:1067] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:56.967198 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.968565 4563 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:56.970973 4563 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:56.968114 4788 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:38:56.957651 4862 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:56.974087 4862 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:56.974784 4862 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.976945 4862 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:56.975304 4552 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:56.978677 4876 raft_consensus.cc:979] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:56.904634 4865 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.980320 4625 raft_consensus.cc:1067] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:38:56.980731 2034 leader_election.cc:398] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:56.981204 2034 leader_election.cc:401] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:38:56.981812 4865 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:56.981899 1961 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:56.982658 4879 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:56.983006 4864 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:56.983569 4864 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:56.983273 4879 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:56.984500 4864 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:56.986369 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.986662 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.989408 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:56.991645 1962 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:56.955340 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:56.990065 1994 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:38:56.994477 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:38:56.996117 4872 log.cc:924] Time spent T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20260927 19:38:56.997280 4871 raft_consensus.cc:3032] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:56.997785 4871 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:56.994310 4865 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.007942 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:38:57.007622 4871 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.012315 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:38:56.999934 4879 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:57.017513 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.023260 4792 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:57.024000 4792 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.026512 4792 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:57.028394 4871 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:57.033545 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 67267.9us
W20260927 19:38:57.034173 4791 consensus_meta.cc:287] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.514s user 0.004s sys 0.009s
I20260927 19:38:57.039945 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.040397 4791 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.035351 4877 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:57.042609 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.042923 4877 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:57.043569 4877 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.040035 1962 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.049469 2034 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:57.050464 4878 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:57.050915 4878 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:57.051471 4878 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.057740 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 19:38:57.062238 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 60779.6us
I20260927 19:38:57.063531 4879 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:57.063752 1962 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:57.064180 4879 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:57.064684 4879 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.066987 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.067822 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.091526 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.086205 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.107388 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.094386 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:38:57.080408 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 145717us
I20260927 19:38:57.098235 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.094304 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.080864 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.085117 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.123390 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.107633 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.107204 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.155493 1995 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:38:57.155604 1990 raft_consensus.cc:2396] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 19:38:57.085788 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.073596 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.150653 1996 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:38:57.150758 2141 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:38:57.115305 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.088441 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:57.170945 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:57.085733 4625 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:38:57.157312 1994 raft_consensus.cc:2396] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260927 19:38:57.158447 1987 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:57.091979 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:57.111002 4877 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:57.163056 2108 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.094622 4563 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.061376 4791 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:57.171247 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 82895.2us
I20260927 19:38:57.156404 1989 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:57.127779 4865 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6.
I20260927 19:38:57.170019 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.169256 4763 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:34940 (request call id 479) took 4774 ms. Trace:
I20260927 19:38:57.179569 1995 raft_consensus.cc:2379] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:57.131541 4879 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.173446 4864 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:57.183530 4877 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.183713 4864 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:57.184283 4864 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:57.176743 4882 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:57.089754 2683 raft_consensus.cc:1203] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20260927 19:38:57.180050 4788 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:38:57.193311 4788 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:57.193807 4788 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:38:57.185174 4882 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:57.200500 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:57.243888 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.186199 4878 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.244233 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:57.276005 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 94680.7us
W20260927 19:38:57.315891 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 115514us
W20260927 19:38:57.315943 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 77602.7us
I20260927 19:38:57.244048 4882 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:57.317178 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.318989 1989 raft_consensus.cc:2379] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:57.293895 2686 raft_consensus.cc:1226] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:57.301847 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 257) took 10205 ms. Trace:
I20260927 19:38:57.293534 4846 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.325891 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.348491 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.349187 2144 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:57.348587 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:57.326419 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.349758 2142 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.350322 2137 raft_consensus.cc:2396] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260927 19:38:57.350473 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:57.198443 4563 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.351382 4791 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:57.351830 4791 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:57.352211 4791 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:57.352797 4871 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:57.330072 4869 consensus_queue.cc:1046] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:57.353993 4563 raft_consensus.cc:3032] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:57.354346 4563 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:57.355317 4563 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.356585 4563 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:57.325678 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.329190 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.334518 4878 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.320526 4788 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.358731 2141 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.335593 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.360296 4788 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:38:57.336104 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 94061.1us
I20260927 19:38:57.360224 4791 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.363009 4791 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.363085 4871 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:57.363445 2140 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 9. Current term is 10. Ops: []
I20260927 19:38:57.366154 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.367202 1991 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:57.337535 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.368824 2142 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.372495 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:57.372967 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.344532 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.347097 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.299115 4863 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:57.329913 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.374336 2143 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.180521 4763 rpcz_store.cc:276] 0927 19:38:52.394848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:52.395404 (+ 556us) service_pool.cc:227] Handling call
0927 19:38:52.625232 (+229828us) ts_tablet_manager.cc:928] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Initiating tablet copy from peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
0927 19:38:55.778571 (+3153339us) tablet_copy_client.cc:361] Tablet copy session begun
0927 19:38:56.893131 (+1114560us) tablet_metadata.cc:641] Metadata flushed
0927 19:38:56.948891 (+ 55760us) tablet_copy_client.cc:490] Wrote new tablet metadata
0927 19:38:57.168342 (+219451us) tablet_copy_client.cc:411] Wrote new consensus metadata
0927 19:38:57.168674 (+ 332us) tablet_replica.cc:205] Creating consensus instance
0927 19:38:57.169047 (+ 373us) ts_tablet_manager.cc:983] Replica registered.
0927 19:38:57.169144 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":90222,"tablet-copy.queue_time_us":201076,"thread_start_us":243784,"threads_started":2}
I20260927 19:38:57.376691 4863 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.341499 4879 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:57.368649 4792 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:38:57.373250 2679 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.378803 4792 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:57.379346 4792 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:38:57.381711 2143 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.381995 4863 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.342999 4882 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:57.384409 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 46934.1us
I20260927 19:38:57.385306 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.251866 1893 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260927 19:38:57.571980 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 186746us
I20260927 19:38:57.413743 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.445828 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.322515 1891 rpcz_store.cc:276] 0927 19:38:47.096633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:47.097504 (+ 871us) service_pool.cc:227] Handling call
0927 19:38:57.301594 (+10204090us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:54.471781 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:54.659704 (+187923us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:54.659743 (+ 39us) write_op.cc:435] Acquired schema lock
0927 19:38:54.660220 (+ 477us) tablet.cc:660] Decoding operations
0927 19:38:54.702732 (+ 42512us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:54.702788 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:54.702818 (+ 30us) tablet.cc:683] Acquiring locks for 10 operations
0927 19:38:54.744058 (+ 41240us) tablet.cc:699] Row locks acquired
0927 19:38:54.744083 (+ 25us) write_op.cc:260] PREPARE: finished
0927 19:38:54.744296 (+ 213us) write_op.cc:270] Start()
0927 19:38:54.744445 (+ 149us) write_op.cc:276] Timestamp: P: 1790537934744230 usec, L: 0
0927 19:38:54.744468 (+ 23us) op_driver.cc:331] REPLICATION: starting
0927 19:38:54.745312 (+ 844us) log.cc:842] Serialized 4009 byte log entry
0927 19:38:56.880341 (+2135029us) op_driver.cc:444] REPLICATION: finished
0927 19:38:56.996201 (+115860us) write_op.cc:301] APPLY: starting
0927 19:38:56.996377 (+ 176us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:56.996766 (+ 389us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:56.996970 (+ 204us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:57.095013 (+ 98043us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:57.095361 (+ 348us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0927 19:38:57.095403 (+ 42us) write_op.cc:312] APPLY: finished
0927 19:38:57.095937 (+ 534us) log.cc:842] Serialized 77 byte log entry
0927 19:38:57.096146 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:57.096348 (+ 202us) write_op.cc:454] Released schema lock
0927 19:38:57.096431 (+ 83us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:57.096810 (+ 379us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:57.096850 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3164789,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":115247,"apply.run_cpu_time_us":9344,"apply.run_wall_time_us":101121,"num_ops":10,"prepare.queue_time_us":527771,"prepare.run_cpu_time_us":55729,"prepare.run_wall_time_us":1765969,"raft.queue_time_us":205012,"raft.run_cpu_time_us":47825,"raft.run_wall_time_us":218993,"replication_time_us":1993005,"thread_start_us":1345489,"threads_started":4,"wal-append.queue_time_us":1390130}]]}
I20260927 19:38:57.413880 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:57.573458 1962 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:57.389732 2654 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:57.573336 1995 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:38:57.579237 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.580195 1962 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:57.581584 4876 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:57.456717 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.584456 1989 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:57.401299 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.587275 2140 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.592881 4878 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:57.593474 4878 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:57.593928 4878 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.596971 4625 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:57.599685 1893 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:57.588882 4877 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 19:38:57.606007 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 192352us
I20260927 19:38:57.611975 4865 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:57.612756 4865 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:57.613346 4865 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:57.612012 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:38:57.616608 4878 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:57.580690 4882 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.620899 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.610982 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.528506 4846 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:57.627379 2145 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.385568 4763 tablet_copy_client.cc:791] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 0 data blocks...
I20260927 19:38:57.633623 4456 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:57.634284 4456 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:57.634850 4456 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:57.639938 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.413689 4871 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.217s
I20260927 19:38:57.627712 4874 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:57.443903 4864 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.647976 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.654979 1990 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 7. Current term is 8. Ops: []
I20260927 19:38:57.644984 4874 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:38:57.679430 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 92875.4us
I20260927 19:38:57.647991 2679 raft_consensus.cc:2454] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:38:57.662370 4763 tablet_copy_client.cc:655] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Starting download of 1 WAL segments...
I20260927 19:38:57.684441 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.654369 4876 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:57.681965 4874 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:38:57.687618 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 302293us
W20260927 19:38:57.692659 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 112525us
I20260927 19:38:57.691642 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.695322 4864 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:57.697549 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.698328 1996 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:57.698521 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.699836 2654 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:57.697971 2034 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:57.701129 1962 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.698025 1993 raft_consensus.cc:1226] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:57.686609 4876 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:57.702729 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.703927 1988 raft_consensus.cc:2396] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260927 19:38:57.705339 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.707438 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.712302 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.712729 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.713408 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.714003 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.715206 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:57.716804 2142 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 19:38:57.691269 4792 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.721558 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.720973 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.723135 4792 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.709867 4887 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260927 19:38:57.723387 4456 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.728123 4456 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.726373 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.725709 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.707561 4863 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:57.729777 4791 raft_consensus.cc:2781] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:57.736837 4791 raft_consensus.cc:686] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:57.737079 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.737614 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.737735 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.738473 1993 raft_consensus.cc:2379] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:57.738544 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.739137 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.739250 4791 consensus_queue.cc:237] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.739985 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.710753 4325 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:57.741053 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.741549 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.742321 1992 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:38:57.743691 2136 raft_consensus.cc:1261] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260927 19:38:57.732983 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.744493 4879 raft_consensus.cc:1050] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:57.746089 2142 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:57.746654 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.746737 4879 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.747697 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.747943 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.741631 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.748008 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:57.732056 2140 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.750216 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.750799 4882 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:57.751233 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.751610 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.751408 4882 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.751930 1990 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:57.752239 1994 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:57.753124 2136 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:57.753432 4882 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:57.753821 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.754774 1992 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:57.755386 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.755990 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.756989 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.750823 2139 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:57.757254 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.764976 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.767088 4878 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.768765 4882 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:57.734901 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.771571 1989 raft_consensus.cc:2379] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
W20260927 19:38:57.777344 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 71917.2us
I20260927 19:38:57.779559 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.780594 1962 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.782799 2139 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:38:57.769769 4882 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.786811 1995 raft_consensus.cc:1261] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260927 19:38:57.790247 4456 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:57.790511 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:57.786254 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4977621330000375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
W20260927 19:38:57.791186 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4347621329999356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
I20260927 19:38:57.790982 4456 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:57.790302 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.793331 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.794047 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.794919 1991 raft_consensus.cc:2362] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5. Re-sending same reply.
I20260927 19:38:57.795535 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.796851 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:57.795579 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:38:57.791555 2037 connection.cc:232] RPC call timeout handler was delayed by -1.4077621330000056s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260927 19:38:57.800052 2037 connection.cc:232] RPC call timeout handler was delayed by -1.3330621329999985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20260927 19:38:57.800336 2037 connection.cc:232] RPC call timeout handler was delayed by -1.2493621329999769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
W20260927 19:38:57.800617 2037 connection.cc:232] RPC call timeout handler was delayed by -1.200762133000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20260927 19:38:57.800989 2037 connection.cc:232] RPC call timeout handler was delayed by -1.0621621330000153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2013s for a response
W20260927 19:38:57.805958 2037 connection.cc:232] RPC call timeout handler was delayed by -1.0099621330000446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
I20260927 19:38:57.812744 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.789498 4871 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:57.820607 4882 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:57.802119 4863 raft_consensus.cc:686] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:57.736330 4865 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.823891 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.823910 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.802469 4456 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:57.810379 4877 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:57.831032 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:57.838416 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.840010 1962 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.840338 4877 raft_consensus.cc:686] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:57.843036 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.843420 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.844045 4877 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:57.844477 4863 consensus_queue.cc:237] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.844089 2062 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.847440 4869 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 19:38:57.836688 2679 raft_consensus.cc:1226] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:57.855940 1962 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.844471 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.860016 4887 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:57.860261 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.860479 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.860606 4887 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:57.860976 2062 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:57.862959 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.861951 4887 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.868819 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.865826 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.843086 4878 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:57.837316 4871 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:57.884812 2064 raft_consensus.cc:1226] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:57.884943 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.882045 2068 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:57.886070 2063 raft_consensus.cc:1226] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:57.882359 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:57.888545 4871 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.857096 4869 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:57.893904 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.901232 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:57.914131 2679 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 9. Current term is 10. Ops: []
I20260927 19:38:57.919728 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:57.918671 2111 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:57.922782 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.883813 4878 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:57.924973 2063 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:57.937109 2139 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:57.932876 4878 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:57.974290 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.978533 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.982065 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:57.983400 2070 raft_consensus.cc:2379] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:57.943941 4625 consensus_queue.cc:1046] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260927 19:38:57.999161 4865 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:57.946059 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:38:58.002624 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 56656.3us
I20260927 19:38:58.003980 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.009910 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.010334 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.011286 2063 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:58.014940 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.014945 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.016825 2064 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:58.015770 2066 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:38:58.017679 4901 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:38:57.974185 4461 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260927 19:38:58.022930 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 188514us
I20260927 19:38:58.024312 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.025811 4903 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:38:58.025952 2654 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:58.026892 4792 raft_consensus.cc:3032] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:38:58.027299 4792 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:58.034931 1991 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260927 19:38:58.036075 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.035988 4896 consensus_queue.cc:1046] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:38:58.037076 2066 raft_consensus.cc:2379] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:58.037302 4896 raft_consensus.cc:3032] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:58.037724 4896 raft_consensus.cc:729] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:58.031881 4889 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:58.040180 4889 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:58.040673 4889 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.015802 4874 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.051569 4874 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:38:58.052351 2686 raft_consensus.cc:1226] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:58.023168 4461 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:58.053933 2141 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 9. Current term is 10. Ops: []
I20260927 19:38:58.060163 2683 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 LEARNER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:38:58.062973 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:58.038622 4896 consensus_queue.cc:260] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.064671 4896 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.021598 4876 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:58.066093 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 44205.2us
I20260927 19:38:58.053766 4461 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.028836 4910 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:58.069454 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:58.073050 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 98232.2us
I20260927 19:38:58.079198 2066 raft_consensus.cc:2379] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:58.082064 1989 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:58.082650 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.083642 1994 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260927 19:38:58.083305 2070 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:58.065654 4791 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260927 19:38:58.090337 4878 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.092935 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.093894 2070 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:58.095803 4878 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:58.098214 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:58.099036 2141 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:58.028117 4792 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.104660 4869 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.103323 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:58.186414 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 103894us
I20260927 19:38:58.297885 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.155118 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.304824 2064 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:58.113797 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:58.154609 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.307606 2136 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.107215 4876 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:58.306883 4901 consensus_queue.cc:1159] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "69f09866d0a1477585f05b2f82ff4c67" responder_term: 10 status { last_received { term: 9 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260927 19:38:58.311486 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:58.313022 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:58.156227 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:58.284251 1893 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 156) took 14786 ms (client timeout 15000 ms). Trace:
W20260927 19:38:58.284538 4907 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.275s user 0.002s sys 0.002s
I20260927 19:38:58.109164 4889 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:58.290009 4917 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260927 19:38:58.102478 4910 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:58.313942 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.306281 2141 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:38:58.271706 4906 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:58.315681 2070 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
W20260927 19:38:58.178000 4912 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.165s user 0.005s sys 0.000s
I20260927 19:38:58.319062 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.338490 2062 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:58.238966 4461 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.320724 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:58.320500 2068 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:58.299975 4792 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.320498 2067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:38:58.321995 2686 raft_consensus.cc:1226] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:58.321312 2071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
W20260927 19:38:58.319599 1893 rpcz_store.cc:269] 0927 19:38:43.498187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:43.498375 (+ 188us) service_pool.cc:227] Handling call
0927 19:38:44.491911 (+993536us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54002f58d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79198bcd6c 00007f79199b6c3e 00007f7917cc8fb4 00007f7917cc8f42 00007f7917cc8e71 00007f7917cc8dec 00007f7917cc80a2 00007f7911d8c94c 00007f7911d8bd86
0927 19:38:58.284159 (+13792248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:57.674720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:57.675457 (+ 737us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:57.675530 (+ 73us) write_op.cc:435] Acquired schema lock
0927 19:38:57.675557 (+ 27us) tablet.cc:660] Decoding operations
0927 19:38:57.676183 (+ 626us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:57.676318 (+ 135us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:57.676374 (+ 56us) tablet.cc:683] Acquiring locks for 2 operations
0927 19:38:57.677043 (+ 669us) tablet.cc:699] Row locks acquired
0927 19:38:57.677092 (+ 49us) write_op.cc:260] PREPARE: finished
0927 19:38:57.677313 (+ 221us) write_op.cc:270] Start()
0927 19:38:57.677505 (+ 192us) write_op.cc:276] Timestamp: P: 1790537937677229 usec, L: 0
0927 19:38:57.677537 (+ 32us) op_driver.cc:331] REPLICATION: starting
0927 19:38:57.678383 (+ 846us) log.cc:842] Serialized 919 byte log entry
0927 19:38:57.815319 (+136936us) op_driver.cc:444] REPLICATION: finished
0927 19:38:57.862043 (+ 46724us) write_op.cc:301] APPLY: starting
0927 19:38:57.862130 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0927 19:38:57.862239 (+ 109us) tablet.cc:1368] finished BulkCheckPresence
0927 19:38:57.862347 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:38:57.910236 (+ 47889us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:38:57.933006 (+ 22770us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0927 19:38:57.942282 (+ 9276us) write_op.cc:312] APPLY: finished
0927 19:38:57.973519 (+ 31237us) log.cc:842] Serialized 29 byte log entry
0927 19:38:58.016778 (+ 43259us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:58.016873 (+ 95us) write_op.cc:454] Released schema lock
0927 19:38:58.016923 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0927 19:38:58.017200 (+ 277us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:38:58.017227 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2250,"reported_tablets":20,"spinlock_wait_cycles":610037632,"child_traces":[["op",{"apply.queue_time_us":44024,"apply.run_cpu_time_us":3592,"apply.run_wall_time_us":157987,"num_ops":2,"prepare.queue_time_us":60620,"prepare.run_cpu_time_us":4070,"prepare.run_wall_time_us":4069,"replication_time_us":137664,"thread_start_us":72015,"threads_started":2}]]}
I20260927 19:38:58.307546 4863 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:58.346041 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:58.337909 4869 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:38:58.351585 2064 raft_consensus.cc:2379] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:58.349057 4456 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260927 19:38:58.355768 2071 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:58.356181 2071 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.358265 4863 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.361158 2067 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:38:58.361817 2679 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:38:58.339700 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.337859 2066 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:58.361653 2067 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.366693 4922 consensus_queue.cc:1046] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:38:58.347069 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.372725 4901 raft_consensus.cc:3032] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:58.373329 4901 raft_consensus.cc:729] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:58.373427 4889 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:58.374707 4901 consensus_queue.cc:260] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.376605 4901 raft_consensus.cc:3037] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.145726 4914 consensus_queue.cc:1046] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.225s
I20260927 19:38:58.379313 4914 raft_consensus.cc:3032] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:38:58.379787 4914 raft_consensus.cc:729] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:58.347312 4791 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260927 19:38:58.368386 4906 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.390786 4910 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:58.381295 4914 consensus_queue.cc:260] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.428262 2062 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
W20260927 19:38:58.433470 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 130400us
W20260927 19:38:58.438485 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 91506.7us
I20260927 19:38:58.440749 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:58.441493 1990 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
W20260927 19:38:58.439312 4919 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.111s user 0.000s sys 0.003s
I20260927 19:38:58.376709 1890 catalog_manager.cc:5495] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.432219 4461 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:58.442068 2111 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.452090 2136 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:38:58.455626 2111 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:58.456974 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 113552us
I20260927 19:38:58.457525 1890 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:58.432931 4791 consensus_queue.cc:1046] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260927 19:38:58.461208 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.464263 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:58.457116 4788 raft_consensus.cc:2741] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260927 19:38:58.470283 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.460857 2684 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:38:58.461562 4863 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:58.459724 4906 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.486789 1890 catalog_manager.cc:5495] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 3 to 4, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 4 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.490131 2686 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.490351 2141 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:58.489097 1890 catalog_manager.cc:5495] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.492971 1890 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:58.493435 2071 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.469635 4896 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:38:58.471029 4921 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:58.495590 4865 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:58.497551 1890 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.500646 4865 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:38:58.504086 2071 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:58.496181 2111 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.505650 4914 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:58.506031 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.502211 1890 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 4, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 4 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.508867 2111 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.512600 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:58.513316 2142 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:58.509546 4461 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:58.510437 2140 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:38:58.514664 4461 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:58.515175 4461 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:58.511350 2141 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260927 19:38:58.519821 1890 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 5, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.517575 4924 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:58.531159 2067 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.535236 2070 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8. Ops: []
I20260927 19:38:58.537053 4871 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:58.537734 1890 catalog_manager.cc:5495] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:38:58.539428 1890 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 8, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 8 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:58.540676 1990 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:38:58.540953 4887 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:58.541431 4887 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.543560 4887 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.545418 1890 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:58.527042 4921 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:58.547292 1890 catalog_manager.cc:5495] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:38:58.549409 1890 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:38:58.550318 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:58.550959 2685 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:58.552093 1959 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:58.554339 4887 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:58.554173 2654 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:58.554852 4887 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:58.555287 4887 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:58.555397 4879 raft_consensus.cc:3032] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:38:58.555790 4879 raft_consensus.cc:729] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:58.534974 2111 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.536268 2067 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:58.564116 1890 catalog_manager.cc:5495] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:58.568400 2141 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:58.556667 4879 consensus_queue.cc:260] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.570325 2111 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.571373 1962 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.565850 4792 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:38:58.574790 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.569504 1890 catalog_manager.cc:5495] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:58.577629 4921 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:58.585839 2111 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:38:58.583468 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 86740.6us
I20260927 19:38:58.588680 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.589145 1890 catalog_manager.cc:5495] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:38:58.595449 2111 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.571682 4928 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260927 19:38:58.574709 4863 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:38:58.602684 2111 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:58.594862 4929 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260927 19:38:58.608711 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.587392 4921 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:58.601763 4910 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:38:58.600446 4792 raft_consensus.cc:686] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:38:58.605672 4896 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:58.603972 4879 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.601250 4924 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:58.613960 1962 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.600131 4864 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260927 19:38:58.611743 4921 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:58.623476 2686 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260927 19:38:58.628747 4792 consensus_queue.cc:237] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.633529 4888 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:38:58.633985 4888 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:58.634116 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:58.634409 4888 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.634490 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:58.635201 4865 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:38:58.635627 4865 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:58.612185 4869 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:58.636039 4865 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:38:58.636548 4869 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.637912 4922 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:38:58.638643 4625 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:38:58.639119 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:38:58.640017 4791 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:58.640429 4791 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:38:58.640841 4791 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:58.635999 4920 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:58.641615 4920 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.643317 4922 consensus_queue.cc:1046] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:38:58.645328 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:58.645881 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:58.622054 4461 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:58.646821 4922 raft_consensus.cc:3032] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:38:58.647270 4922 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:58.643891 4920 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.648144 4922 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.653071 4922 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:58.649081 2110 leader_election.cc:334] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.514s (SENT)
I20260927 19:38:58.621230 4924 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:58.653764 4865 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.609771 4456 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:58.626672 2686 pending_rounds.cc:76] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Aborting all ops after (but not including) 5
I20260927 19:38:58.613529 4928 raft_consensus.cc:2726] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:58.659049 2110 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.626998 4910 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:58.659883 4924 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:58.661195 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.663810 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:58.671388 2684 raft_consensus.cc:2379] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:38:58.672667 4921 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:58.678081 4930 log.cc:924] Time spent T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260927 19:38:58.680104 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:58.680751 2139 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:58.681900 1962 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.682880 4901 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:38:58.683240 4901 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:38:58.683553 4901 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:38:58.685742 4461 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:58.691282 4887 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.614905 4863 raft_consensus.cc:686] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:58.625803 4931 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:58.706434 4863 consensus_queue.cc:237] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.692139 4901 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.709168 4901 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.664328 4869 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.693430 4921 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:38:58.694054 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 45025.9us
I20260927 19:38:58.695341 4788 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:58.695500 4791 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.667333 2686 pending_rounds.cc:97] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.6
I20260927 19:38:58.719079 2686 raft_consensus.cc:2894] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Aborting config change with OpId 5.6: Aborted: Op aborted by new leader
I20260927 19:38:58.622859 4896 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:38:58.692747 4910 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.719784 2686 raft_consensus.cc:2866] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:38:58.721055 4896 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:38:58.724521 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 63417.3us
I20260927 19:38:58.733408 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:58.738425 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:58.739974 4923 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:58.740674 4923 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.743388 4923 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.745265 2140 raft_consensus.cc:2379] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 10: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:38:58.724200 4879 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:38:58.748036 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:58.754935 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:58.722472 4888 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:58.764962 2110 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 3.309s (SENT) [suppressed 62 similar messages]
I20260927 19:38:58.768141 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:58.769290 2684 raft_consensus.cc:2379] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
W20260927 19:38:58.771526 2110 leader_election.cc:334] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.631s (SENT)
I20260927 19:38:58.772018 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.773221 4921 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:38:58.747341 4879 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:38:58.777806 4871 consensus_queue.cc:1035] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260927 19:38:58.739027 4869 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260927 19:38:58.782123 2145 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:38:58.791190 1962 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.801035 2686 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEARNER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:58.781423 4879 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:58.813309 4942 consensus_queue.cc:1046] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:38:58.816972 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:58.817384 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:58.817981 1989 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:38:58.818485 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:38:58.821727 4887 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:58.823334 4888 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:38:58.816218 2110 leader_election.cc:334] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.664s (SENT)
I20260927 19:38:58.826398 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:38:58.828145 2145 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.827342 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:58.806936 4901 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:38:58.833853 4938 consensus_queue.cc:1046] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260927 19:38:58.834460 4461 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:58.763535 4924 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.827589 2111 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.836562 4924 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:58.836709 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:58.837087 4924 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:58.837678 4791 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:58.840551 4763 tablet_copy_client.cc:531] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 19:38:58.840957 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:58.842000 4896 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:58.832510 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.860798 2654 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.814815 4942 raft_consensus.cc:3032] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:58.833055 4865 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:58.853125 2686 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:38:58.884152 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 145618us
I20260927 19:38:58.885313 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.886567 2111 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:38:58.891076 4865 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:58.891567 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:58.861850 4942 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:58.895702 4886 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260927 19:38:58.896302 4942 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.898295 4910 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:58.899641 4931 raft_consensus.cc:3032] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:58.900094 4931 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:58.880257 4846 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:38:58.878661 4901 raft_consensus.cc:686] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:58.898851 4910 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:58.917474 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.918814 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.903429 4461 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:38:58.920336 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:58.897938 4942 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:58.921767 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39d534ed586444279970bb03de60c919"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:58.923736 1992 raft_consensus.cc:595] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:58.927690 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:58.920230 4938 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:38:58.924935 4865 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:38:58.960263 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:58.980304 1992 raft_consensus.cc:595] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:58.980667 1992 raft_consensus.cc:603] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:38:58.936429 4901 consensus_queue.cc:237] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:58.970125 2110 leader_election.cc:334] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.715s (SENT)
I20260927 19:38:58.986639 2110 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:58.930119 4456 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:58.972122 2145 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:58.973773 2686 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:38:58.974926 1989 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:38:58.988152 4946 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:58.990658 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.991814 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.992851 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.013675 1962 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:59.016561 2111 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:38:59.015688 1959 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:58.994401 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.014253 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.018589 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.059058 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:38:59.062790 4921 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:59.067049 4921 raft_consensus.cc:686] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:38:59.067997 4921 consensus_queue.cc:237] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:59.059541 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.901013 4931 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:59.058585 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.070312 4931 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
W20260927 19:38:59.064008 4940 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.059s user 0.002s sys 0.000s
I20260927 19:38:59.075023 4879 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.064482 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.076493 2070 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.020097 2066 raft_consensus.cc:2379] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.059993 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:58.930225 4871 consensus_queue.cc:1046] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260927 19:38:59.019119 2064 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.063611 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:59.019436 4924 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:59.063819 4563 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:38:59.080195 1989 raft_consensus.cc:595] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:38:59.081845 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.079536 2071 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.081239 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.084430 2066 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.082684 2069 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.090682 4910 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.091354 2136 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:38:59.093775 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.094174 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.094700 2070 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.095489 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.094732 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:59.094216 4879 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.098300 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.099026 2062 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.100759 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:59.101742 2686 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:38:59.058957 4946 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:59.102959 1959 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:38:59.114871 4941 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260927 19:38:59.117242 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:38:59.117842 4863 raft_consensus.cc:1050] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:38:59.086786 4563 raft_consensus.cc:686] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:59.130105 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.109795 4888 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:38:59.122978 4896 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:38:59.139860 4909 consensus_queue.cc:1046] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20260927 19:38:59.141016 2066 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.129217 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.146910 4563 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.176975 4916 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:38:59.139032 4888 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:59.198873 4791 raft_consensus.cc:3032] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:38:59.199745 4915 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
W20260927 19:38:59.211752 1959 consensus_peers.cc:487] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 -> Peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629): Couldn't send request to peer 4e0335d1a85f421380029b95a91bf0c4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260927 19:38:59.200421 2110 leader_election.cc:334] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.842s (SENT)
I20260927 19:38:59.239113 2110 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.226603 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.234664 2069 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
W20260927 19:38:59.236289 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 110708us
I20260927 19:38:59.237522 4791 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:38:59.237690 4888 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:59.264464 2062 raft_consensus.cc:1226] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: []
W20260927 19:38:59.266949 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 66585.2us
I20260927 19:38:59.237401 4915 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:38:59.268985 2110 leader_election.cc:334] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.845s (SENT)
I20260927 19:38:59.269636 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.274745 4791 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:38:59.275807 2110 leader_election.cc:334] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.856s (SENT)
I20260927 19:38:59.276105 4896 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:59.276329 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:59.277644 2110 leader_election.cc:334] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.858s (SENT)
I20260927 19:38:59.276146 4791 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:59.262768 4928 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:38:59.278599 2110 leader_election.cc:334] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.907s (SENT)
I20260927 19:38:59.279028 4928 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:59.279060 2110 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.280009 4928 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:38:59.248096 4887 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:59.280653 4887 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.277040 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.281905 2062 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.283309 4887 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.252104 4916 raft_consensus.cc:686] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:38:59.229259 4863 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.304494 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.306785 2063 raft_consensus.cc:2396] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 19:38:59.324285 1990 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:38:59.327570 2136 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:38:59.332952 2136 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:38:59.337764 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:59.203263 4946 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:59.338553 2686 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 10.
I20260927 19:38:59.343835 1959 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:59.345178 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.346364 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:59.347272 2139 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 10.
I20260927 19:38:59.296212 4915 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:38:59.333791 4951 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:59.365543 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 60294.1us
I20260927 19:38:59.355568 4909 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260927 19:38:59.386377 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.395633 1989 raft_consensus.cc:595] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.411106 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.412079 2066 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.412143 2064 raft_consensus.cc:1226] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:38:59.352276 4936 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260927 19:38:59.351455 4945 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260927 19:38:59.416240 2064 raft_consensus.cc:1226] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 3. Current term is 4. Ops: []
I20260927 19:38:59.417104 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.369886 4910 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260927 19:38:59.384056 4909 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:38:59.388063 4869 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:59.418751 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.363377 4916 consensus_queue.cc:237] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:38:59.380789 4951 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:38:59.490741 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 135800us
I20260927 19:38:59.538326 2062 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.426791 2069 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
I20260927 19:38:59.431624 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.397183 4915 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:38:59.567082 4959 log.cc:924] Time spent T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260927 19:38:59.566152 4960 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260927 19:38:59.568353 4763 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap starting.
I20260927 19:38:59.581779 2070 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.596742 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.598809 2071 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.617581 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:59.619189 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:59.596868 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.434144 2067 raft_consensus.cc:2379] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.601876 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.621069 4920 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:59.621568 2069 raft_consensus.cc:2379] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.581343 4869 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.610639 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.612349 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.613983 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.622949 2064 raft_consensus.cc:2379] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:38:59.406687 4942 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:59.622844 2066 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.621626 4920 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.624874 4942 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.628189 4920 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.642021 2063 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
W20260927 19:38:59.628491 2110 leader_election.cc:334] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.997s (SENT)
I20260927 19:38:59.644904 2110 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:38:59.646687 2110 leader_election.cc:334] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.005s (SENT)
I20260927 19:38:59.647342 4920 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:38:59.648814 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.649776 2070 raft_consensus.cc:2396] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 19:38:59.651496 4625 raft_consensus.cc:979] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:59.664336 2110 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.563194 4888 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.683261 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.703171 2071 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.653246 4625 raft_consensus.cc:1067] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:38:59.685145 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:59.685839 4945 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:38:59.713685 4945 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:38:59.716488 4945 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:38:59.689822 2081 tablet_copy_service.cc:342] P 4cc2df048f424e08803a390946004df0: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-db442f51d759423ba3bde8876ae46aa0 received from {username='slave'} at 127.0.0.1:60518
W20260927 19:38:59.690246 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 43612.2us
I20260927 19:38:59.717484 2081 tablet_copy_service.cc:433] P 4cc2df048f424e08803a390946004df0: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-db442f51d759423ba3bde8876ae46aa0 on tablet db442f51d759423ba3bde8876ae46aa0 with peer 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:38:59.718029 2110 leader_election.cc:334] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.029s (SENT)
I20260927 19:38:59.718494 2110 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.719132 2683 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:38:59.719805 4896 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:38:59.595507 4909 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Advancing to term 11
I20260927 19:38:59.694845 2145 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:38:59.711283 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:38:59.711892 4951 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.729516 2145 raft_consensus.cc:595] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:38:59.729800 2686 raft_consensus.cc:2379] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:38:59.727654 2062 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:38:59.731987 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:59.732293 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:38:59.732651 1991 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.734458 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:38:59.735113 1993 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:38:59.737915 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:59.740655 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:38:59.704603 4920 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:38:59.742801 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.748354 2140 raft_consensus.cc:1261] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:38:59.754297 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.755116 2062 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:38:59.728886 4945 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:38:59.781644 2110 leader_election.cc:334] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.030s (SENT)
I20260927 19:38:59.814428 4888 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.742146 4887 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260927 19:38:59.754439 4889 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:38:59.973487 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.909108 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:59.830391 2653 leader_election.cc:334] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.145s (SENT)
I20260927 19:38:59.815500 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:38:59.979622 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:39:00.070338 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 135704us
I20260927 19:38:59.819166 2071 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
W20260927 19:38:59.955758 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 225068us
W20260927 19:38:59.974320 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 192726us
I20260927 19:38:59.885931 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:38:59.886734 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:38:59.934828 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:38:59.906394 4941 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260927 19:38:59.886407 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.098099 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.097765 2067 raft_consensus.cc:2362] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6. Re-sending same reply.
W20260927 19:39:00.082260 1961 leader_election.cc:334] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.549s (SENT)
I20260927 19:38:59.848456 4869 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:38:59.772217 4945 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.176927 1994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:00.178840 1994 raft_consensus.cc:595] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:00.079408 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:38:59.853133 4970 log.cc:924] Time spent T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260927 19:39:00.177547 1961 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.178705 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:00.181730 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:00.101214 4889 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:00.178960 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.185570 2140 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:00.180636 2063 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:38:59.872468 4969 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:00.182391 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:00.195575 2065 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:00.186002 2071 raft_consensus.cc:2362] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4. Re-sending same reply.
W20260927 19:39:00.186142 2110 leader_election.cc:334] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.064s (SENT)
W20260927 19:39:00.183692 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 154715us
I20260927 19:39:00.188154 2139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:39:00.197289 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.197942 2139 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:00.198227 2679 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:00.189106 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:39:00.191673 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.199843 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:00.098878 2686 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:38:59.903676 4942 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.157034 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.211493 2070 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:00.190750 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.069861 4946 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:00.198348 2139 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:00.201295 2108 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:39:00.191694 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 361358us
I20260927 19:39:00.216204 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.212498 2062 raft_consensus.cc:2379] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:00.218713 4909 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.219403 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:00.220804 4909 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.134361 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.191855 4965 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:00.225823 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.233014 2685 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 19:39:00.214632 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 116346us
I20260927 19:39:00.242475 2683 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:00.227324 4946 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.246044 4946 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.246739 2068 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:39:00.230911 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.229048 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.224418 2064 raft_consensus.cc:2379] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:00.225047 2139 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.232800 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:00.262483 1994 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:00.232960 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:00.235206 2653 leader_election.cc:334] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.622s (SENT)
I20260927 19:39:00.264442 2653 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:00.264717 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:00.232738 2063 raft_consensus.cc:1226] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:00.238101 2686 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:00.266470 2139 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.193480 4889 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:00.267215 2144 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:00.277770 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:00.278885 2686 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:00.285624 4909 consensus_queue.cc:1035] T 303e2824ebc54b4c9bce740f15e0105d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:00.288640 2069 raft_consensus.cc:1226] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:00.289971 4876 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:00.291523 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.292421 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:00.293691 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:39:00.294289 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 118620us
I20260927 19:39:00.295802 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.270591 2065 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:00.296945 2067 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:00.308223 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:00.309068 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:00.312749 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:00.323489 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.329937 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:00.265944 4921 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:00.266109 4928 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.295002 4963 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:39:00.295514 4846 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:00.360773 2071 raft_consensus.cc:1261] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:00.373967 2683 raft_consensus.cc:2379] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 7: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:00.375865 2686 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:00.378157 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.215112 4969 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.354231 4921 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:39:00.379942 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 164061us
I20260927 19:39:00.386200 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:39:00.386994 2653 leader_election.cc:334] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.679s (SENT)
I20260927 19:39:00.387483 2653 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:00.388836 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:39:00.378875 4896 consensus_meta.cc:287] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.659s user 0.009s sys 0.006s
I20260927 19:39:00.379532 2108 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:00.390057 1990 raft_consensus.cc:1226] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:00.392201 1993 raft_consensus.cc:1226] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:00.392946 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:00.395839 4874 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:39:00.398254 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:00.353261 4945 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:00.380862 2145 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:00.373317 4846 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:39:00.401518 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 319281us
I20260927 19:39:00.404536 1989 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
W20260927 19:39:00.412823 1961 leader_election.cc:334] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.577s (SENT)
I20260927 19:39:00.408687 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:00.413918 2679 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Advancing to term 11
I20260927 19:39:00.405797 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.400406 4921 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:00.417219 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.418484 1989 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7. Ops: []
W20260927 19:39:00.419826 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 173770us
I20260927 19:39:00.425248 1993 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:00.406639 4896 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:00.431324 2679 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 11.
I20260927 19:39:00.434403 1994 raft_consensus.cc:1261] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:00.411756 4846 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:00.410779 4924 raft_consensus.cc:2781] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:00.452816 2680 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260927 19:39:00.454360 4924 raft_consensus.cc:686] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:00.455592 4924 consensus_queue.cc:237] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.460923 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.461639 2142 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:00.439119 4864 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:39:00.465972 1961 leader_election.cc:334] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.591s (SENT)
I20260927 19:39:00.467911 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:00.468464 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 44042.7us
W20260927 19:39:00.474709 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 87741us
I20260927 19:39:00.475041 1959 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:39:00.470856 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 83223.5us
I20260927 19:39:00.477974 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.478609 2137 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:00.481285 4921 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.482573 2144 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:00.484467 4921 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.487152 2140 raft_consensus.cc:1226] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
W20260927 19:39:00.487541 1961 leader_election.cc:334] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.646s (SENT)
I20260927 19:39:00.487962 1961 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:00.488386 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 193396us
I20260927 19:39:00.488847 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:00.486341 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 49839.8us
W20260927 19:39:00.489180 2110 leader_election.cc:334] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.132s (SENT)
I20260927 19:39:00.490001 2110 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:00.491914 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.492224 4969 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.492372 4968 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:00.492796 4968 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:00.493150 4968 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:00.495160 2142 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:00.498519 4954 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.492249 2143 raft_consensus.cc:1226] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [2.2-5.3]
I20260927 19:39:00.500315 2140 raft_consensus.cc:1226] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:00.500422 1959 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:39:00.475973 2653 leader_election.cc:334] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.824s (SENT)
I20260927 19:39:00.503289 2653 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.505908 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:00.506819 4957 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:00.507227 2143 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:00.511745 2136 raft_consensus.cc:1226] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:00.513806 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:00.516371 2110 leader_election.cc:334] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.179s (SENT)
I20260927 19:39:00.513168 2137 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
W20260927 19:39:00.517946 2110 leader_election.cc:334] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.531s (SENT)
I20260927 19:39:00.518381 2110 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:39:00.519518 1961 leader_election.cc:334] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.666s (SENT)
I20260927 19:39:00.525324 2137 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:39:00.527585 4896 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:39:00.576918 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 58231.7us
I20260927 19:39:00.517868 4864 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:00.616667 1959 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:39:00.580396 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 52862.4us
I20260927 19:39:00.525818 4921 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.581584 4791 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:39:00.627156 4975 log.cc:924] Time spent T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260927 19:39:00.628405 4563 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 FOLLOWER]: Leader election won for term 11
I20260927 19:39:00.628984 4563 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:00.587900 4906 raft_consensus.cc:979] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:00.590376 4979 raft_consensus.cc:979] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:00.519174 4871 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:00.529078 4980 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:00.594578 4916 consensus_queue.cc:1046] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:00.597246 4910 consensus_queue.cc:1046] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260927 19:39:00.598902 4976 log.cc:924] Time spent T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.179s user 0.002s sys 0.000s
I20260927 19:39:00.597669 4968 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.599726 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.680701 2144 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Advancing to term 11
I20260927 19:39:00.601930 2137 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 19:39:00.604328 4978 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20260927 19:39:00.610852 1989 raft_consensus.cc:1261] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:39:00.610858 2140 raft_consensus.cc:1261] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20260927 19:39:00.470853 4909 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260927 19:39:00.625151 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.626663 4791 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.642310 4981 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:00.629837 4563 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.634192 2139 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260927 19:39:00.634004 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.635965 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:39:00.637643 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 58751.8us
I20260927 19:39:00.679925 2680 raft_consensus.cc:1226] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
W20260927 19:39:00.687063 1961 leader_election.cc:334] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.761s (SENT)
W20260927 19:39:00.587258 2037 connection.cc:232] RPC call timeout handler was delayed by -1.2317434099999218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260927 19:39:00.695451 2037 connection.cc:232] RPC call timeout handler was delayed by -1.2038434099999904s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20260927 19:39:00.695962 2037 connection.cc:232] RPC call timeout handler was delayed by -1.1183434099999658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260927 19:39:00.696305 2037 connection.cc:232] RPC call timeout handler was delayed by -1.1111434099999542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20260927 19:39:00.668442 4974 raft_consensus.cc:1067] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:00.695451 1961 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:00.697306 1993 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 7: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:00.679584 4922 raft_consensus.cc:1067] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
W20260927 19:39:00.696916 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 55882.5us
I20260927 19:39:00.703233 2037 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.643255 4946 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:39:00.706702 1961 leader_election.cc:334] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.775s (SENT)
I20260927 19:39:00.707388 1961 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:00.708755 1961 leader_election.cc:334] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.808s (SENT)
I20260927 19:39:00.679829 4980 raft_consensus.cc:686] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:00.643810 4625 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260927 19:39:00.709311 1961 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:00.719671 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 81667.1us
I20260927 19:39:00.663563 4871 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:00.738664 1961 leader_election.cc:334] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.848s (SENT)
I20260927 19:39:00.739228 1961 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.715567 4896 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:00.762859 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2508) took 1023 ms. Trace:
I20260927 19:39:00.764926 2145 raft_consensus.cc:1203] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[6.4-6.4] Dedup: 6.4->[]
I20260927 19:39:00.765628 2145 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260927 19:39:00.762282 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 74373.5us
I20260927 19:39:00.763288 2141 rpcz_store.cc:276] 0927 19:38:59.739717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:59.741450 (+ 1733us) service_pool.cc:227] Handling call
0927 19:39:00.071089 (+329639us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:39:00.147092 (+ 76003us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:00.147137 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:39:00.189795 (+ 42658us) log.cc:842] Serialized 52 byte log entry
0927 19:39:00.383719 (+193924us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:39:00.466830 (+ 83111us) log.cc:842] Serialized 14 byte log entry
0927 19:39:00.467051 (+ 221us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:00.467118 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:00.603713 (+136595us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:00.759727 (+156014us) raft_consensus.cc:1671] finished
0927 19:39:00.762620 (+ 2893us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:00.762797 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107993,"prepare.queue_time_us":89888,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":109913,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":14641,"spinlock_wait_cycles":81431808,"thread_start_us":238731,"threads_started":2,"wal-append.queue_time_us":217573}
W20260927 19:39:00.767421 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 70487us
I20260927 19:39:00.750077 4942 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.718788 4946 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:00.719373 4846 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.770246 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:00.761819 4922 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.780297 4625 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.769161 4920 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:39:00.856791 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 86689.1us
W20260927 19:39:00.787281 4991 log.cc:924] Time spent T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260927 19:39:00.775688 4910 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:39:00.758333 4896 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.771588 4915 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:39:00.719616 4928 consensus_queue.cc:1046] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260927 19:39:00.783457 4923 consensus_queue.cc:1046] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260927 19:39:00.877619 4920 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:00.884544 4994 raft_consensus.cc:3032] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:00.884738 4928 raft_consensus.cc:3032] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:00.884981 4994 raft_consensus.cc:729] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:00.885237 4928 raft_consensus.cc:729] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:00.885531 4920 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:00.781930 4942 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:00.889623 4942 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.892045 4942 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.699560 4981 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:00.864676 2683 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 7: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:00.875285 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:00.864477 4978 log.cc:924] Time spent T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260927 19:39:00.893929 4846 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:00.858217 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 73491.3us
I20260927 19:39:00.882225 5000 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:00.895144 4981 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:00.896432 4968 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.896678 4887 raft_consensus.cc:3032] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:00.897061 4887 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:39:00.898554 4889 consensus_meta.cc:287] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.586s user 0.007s sys 0.003s
I20260927 19:39:00.899029 4889 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.899319 4928 consensus_queue.cc:260] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:00.900848 4928 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:00.905902 4871 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.882777 4946 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:00.918095 2037 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.882966 4896 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:00.883517 4988 log.cc:924] Time spent T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
I20260927 19:39:00.770790 4980 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:00.917270 4871 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:00.886024 4994 consensus_queue.cc:260] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:00.915934 2144 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 11.
I20260927 19:39:00.901073 4889 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:00.918723 2110 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:00.930940 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:39:00.919314 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 60266.7us
I20260927 19:39:00.920624 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.946018 1993 raft_consensus.cc:3032] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:00.946441 1993 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:00.920744 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.946976 2110 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:00.944342 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.922245 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:00.868579 4921 consensus_queue.cc:1046] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260927 19:39:00.932314 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.949555 2679 raft_consensus.cc:2354] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:00.948941 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.935261 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.951073 2108 leader_election.cc:398] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:00.951541 2108 leader_election.cc:401] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:00.944682 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.953012 2144 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:00.943078 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.955858 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:39:00.957270 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 218650us
W20260927 19:39:00.957923 1961 leader_election.cc:334] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.882s (SENT)
I20260927 19:39:00.958930 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:00.958284 1961 leader_election.cc:302] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.958925 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:00.959885 1996 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:00.960280 1995 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 7: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
W20260927 19:39:00.960110 1961 leader_election.cc:334] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.885s (SENT)
I20260927 19:39:00.960733 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:00.962284 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:00.962889 2141 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 9
I20260927 19:39:00.924234 4928 consensus_queue.cc:1046] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260927 19:39:00.929325 5000 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:00.955134 1993 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:00.954131 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 291359us
I20260927 19:39:00.964718 4791 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:00.966784 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.973068 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:00.955034 4896 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:00.974049 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:00.974679 1993 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:00.960137 4974 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.007912 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.008843 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:01.009615 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:01.014299 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.016225 2140 raft_consensus.cc:2379] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:01.016520 2144 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
W20260927 19:39:01.014578 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54497.5us
W20260927 19:39:01.019025 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 46000.9us
W20260927 19:39:01.019099 1961 leader_election.cc:334] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.885s (SENT)
W20260927 19:39:01.026216 1961 leader_election.cc:334] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.894s (SENT)
I20260927 19:39:01.026854 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.028333 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:00.965024 4998 raft_consensus.cc:2726] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:01.007513 4625 consensus_queue.cc:1046] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260927 19:39:00.975806 4946 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:39:01.043177 1961 leader_election.cc:334] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.964s (SENT)
I20260927 19:39:01.047381 1961 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.029129 4910 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260927 19:39:01.052663 5007 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:01.053440 5001 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.029403 4906 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:01.042465 4791 consensus_queue.cc:1035] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260927 19:39:01.026594 4864 consensus_queue.cc:1035] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260927 19:39:01.045666 4871 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:01.065328 4887 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.068626 1993 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:01.041391 5000 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.078073 2141 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:01.079555 1962 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:01.085403 1962 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.086508 4979 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:01.086975 4979 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:01.087100 4888 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.087440 4979 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:01.079851 1987 raft_consensus.cc:3032] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:01.092936 1987 raft_consensus.cc:729] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:01.081573 4915 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:39:01.084090 1961 leader_election.cc:334] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.036s (SENT)
I20260927 19:39:01.095295 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:01.101440 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 51794.2us
I20260927 19:39:01.109587 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:01.022429 4994 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:01.118906 4996 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:01.125806 1987 consensus_queue.cc:260] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.127408 1987 raft_consensus.cc:3037] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:01.133741 4792 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:01.134431 4792 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:01.135603 4792 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.143314 2683 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260927 19:39:01.146595 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.146932 4924 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260927 19:39:01.045312 4916 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:39:01.163146 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 79100.7us
I20260927 19:39:01.172467 4906 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:39:01.170190 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 141962us
I20260927 19:39:01.183949 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.189755 1987 raft_consensus.cc:2454] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:01.170060 4910 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.174815 4625 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:01.193195 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:01.200373 4977 log.cc:924] Time spent T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.768s user 0.005s sys 0.001s
I20260927 19:39:01.202145 4871 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.202641 4979 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.194290 4946 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.212680 4625 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:01.216827 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:01.222654 2145 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:01.226022 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.227849 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.232949 1996 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:01.244879 2145 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.213848 4979 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:01.246466 5012 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.247320 5006 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.248071 4871 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
W20260927 19:39:01.249454 4876 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260927 19:39:01.252696 4871 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:01.253172 4461 raft_consensus.cc:3032] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:39:01.253676 4461 raft_consensus.cc:729] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:01.254734 4461 consensus_queue.cc:260] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.256300 4461 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:01.258466 4995 raft_consensus.cc:3032] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:01.258901 4995 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:01.259693 4995 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.263096 4625 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:01.253206 4871 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:01.264828 4887 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:01.265195 4871 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.266939 4763 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260927 19:39:01.267956 4763 tablet_bootstrap.cc:492] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Bootstrap complete.
I20260927 19:39:01.269059 4763 ts_tablet_manager.cc:1397] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Time spent bootstrapping tablet: real 1.774s user 0.027s sys 0.038s
I20260927 19:39:01.267885 4871 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:01.279984 2143 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:01.280701 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.281464 2137 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
W20260927 19:39:01.306684 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 142983us
I20260927 19:39:01.317821 1962 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:01.322878 2111 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:01.323966 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 90123.1us
W20260927 19:39:01.325124 2110 leader_election.cc:334] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.140s (SENT)
W20260927 19:39:01.325841 1961 leader_election.cc:334] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.077s (SENT)
I20260927 19:39:01.325773 2110 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:39:01.326678 1961 leader_election.cc:334] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.087s (SENT)
I20260927 19:39:01.327189 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.329571 4923 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:39:01.343842 1961 leader_election.cc:334] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.145s (SENT)
I20260927 19:39:01.344528 1961 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.348019 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:01.331059 5009 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260927 19:39:01.350308 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 113363us
I20260927 19:39:01.331666 4625 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:01.353716 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.354658 4946 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:01.329962 4896 consensus_queue.cc:1035] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260927 19:39:01.356352 4946 consensus_queue.cc:1046] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:39:01.357558 4942 raft_consensus.cc:3032] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:01.357882 4942 raft_consensus.cc:729] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:01.358649 4942 consensus_queue.cc:260] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:01.353888 1961 leader_election.cc:334] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.186s (SENT)
I20260927 19:39:01.360349 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.351723 4979 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:01.366835 4625 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:01.363858 4998 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:01.381951 5000 consensus_queue.cc:1035] T 3c4fa5fe3da244bb9e845557555810b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:39:01.382465 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:01.383404 1961 leader_election.cc:334] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.216s (SENT)
I20260927 19:39:01.372891 4942 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:01.383805 4954 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.388159 4763 raft_consensus.cc:348] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.389724 5006 consensus_queue.cc:1046] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:39:01.411547 2145 raft_consensus.cc:1226] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:01.413589 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.363386 4864 raft_consensus.cc:2726] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260927 19:39:01.413884 1961 leader_election.cc:334] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.238s (SENT)
I20260927 19:39:01.420248 1961 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.420125 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:01.423785 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.425129 5021 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:01.426957 1893 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:01.429162 1893 catalog_manager.cc:5495] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 5 to 6, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 6 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:01.412393 4998 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:01.434271 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.413151 5000 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:01.435063 1987 raft_consensus.cc:2354] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:01.437656 1993 raft_consensus.cc:1261] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 9. (index mismatch)
I20260927 19:39:01.438410 5021 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:01.438724 5032 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:01.442694 2111 leader_election.cc:398] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:01.444473 4461 consensus_queue.cc:1035] T 6ec895e575154c488c4fb6666ae75ce3 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 19:39:01.446033 5032 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.422600 5012 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:01.432734 5002 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:01.449887 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.448064 2139 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 5 to 7, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:01.427275 5006 consensus_queue.cc:1225] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer 3d1bc2cd9efe43399ddba396f5ddff85 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260927 19:39:01.437716 4998 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:01.457261 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.457041 5021 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.463694 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
W20260927 19:39:01.464635 1961 leader_election.cc:334] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.548s (SENT)
I20260927 19:39:01.465238 1961 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.468226 2068 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:39:01.468112 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.432986 4886 consensus_queue.cc:1046] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260927 19:39:01.469841 4999 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:01.470049 4886 raft_consensus.cc:3032] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:01.470491 4886 raft_consensus.cc:729] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:01.470516 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.470829 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.471421 2068 raft_consensus.cc:2379] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:01.471462 4886 consensus_queue.cc:260] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.472375 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.472780 4886 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:01.472673 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.473621 2067 raft_consensus.cc:2396] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260927 19:39:01.469904 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.474982 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.475173 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.476101 2062 raft_consensus.cc:2379] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:01.477079 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:01.478036 5018 log.cc:924] Time spent T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.126s user 0.002s sys 0.000s
I20260927 19:39:01.479040 2063 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260927 19:39:01.479415 2063 pending_rounds.cc:76] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 5
I20260927 19:39:01.479625 2063 pending_rounds.cc:97] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.6
I20260927 19:39:01.479862 2063 raft_consensus.cc:2894] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Aborting config change with OpId 5.6: Aborted: Op aborted by new leader
I20260927 19:39:01.480322 2063 raft_consensus.cc:2866] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:01.480921 4763 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEARNER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Initialized, Role: LEARNER
I20260927 19:39:01.474553 2066 raft_consensus.cc:2454] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.481691 4763 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:01.484467 4763 ts_tablet_manager.cc:1428] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Time spent starting tablet: real 0.174s user 0.002s sys 0.006s
I20260927 19:39:01.486613 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.412449 4625 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.494019 2155 tablet_copy_service.cc:342] P 69f09866d0a1477585f05b2f82ff4c67: Request end of tablet copy session 4e0335d1a85f421380029b95a91bf0c4-51dd0aa486b149cb9fc4a9dd54e3cdf4 received from {username='slave'} at 127.0.0.1:46930
I20260927 19:39:01.494516 2155 tablet_copy_service.cc:433] P 69f09866d0a1477585f05b2f82ff4c67: ending tablet copy session 4e0335d1a85f421380029b95a91bf0c4-51dd0aa486b149cb9fc4a9dd54e3cdf4 on tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 with peer 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:01.500403 2137 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 11 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260927 19:39:01.506975 4864 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260927 19:39:01.510794 2062 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:01.514143 4791 raft_consensus.cc:3032] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:01.514552 4791 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:01.515388 4791 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.516359 4995 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:01.516670 4791 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:01.518194 4962 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 5 to 7, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:01.521979 2064 raft_consensus.cc:2396] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260927 19:39:01.524020 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.524700 2068 raft_consensus.cc:2454] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:01.527248 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.527192 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.528831 2066 raft_consensus.cc:2396] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260927 19:39:01.538983 1989 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:01.547753 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.527102 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 236) took 8665 ms. Trace:
I20260927 19:39:01.533219 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.551334 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.551967 2068 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.552788 2067 raft_consensus.cc:2379] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 5: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:01.556627 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.557531 2063 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
W20260927 19:39:01.560102 1961 leader_election.cc:334] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.504s (SENT)
I20260927 19:39:01.560657 1961 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.562899 2071 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:01.530140 1893 catalog_manager.cc:5495] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 7 to 8, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 8 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:01.565346 1893 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 2 to 3, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 3 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:01.567032 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.569209 1893 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:01.570915 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:01.572027 1961 leader_election.cc:334] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.630s (SENT)
I20260927 19:39:01.572603 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.574002 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.575379 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.548822 1890 rpcz_store.cc:276] 0927 19:38:52.861989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:52.878272 (+ 16283us) service_pool.cc:227] Handling call
0927 19:39:01.527007 (+8648735us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:38:58.818679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:38:58.819326 (+ 647us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:38:58.819368 (+ 42us) write_op.cc:435] Acquired schema lock
0927 19:38:58.819391 (+ 23us) tablet.cc:660] Decoding operations
0927 19:38:58.821070 (+ 1679us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:38:58.821175 (+ 105us) write_op.cc:641] Partition lock acquired for write op
0927 19:38:58.821211 (+ 36us) tablet.cc:683] Acquiring locks for 16 operations
0927 19:38:58.895178 (+ 73967us) tablet.cc:699] Row locks acquired
0927 19:38:58.895204 (+ 26us) write_op.cc:260] PREPARE: finished
0927 19:38:58.906407 (+ 11203us) write_op.cc:270] Start()
0927 19:38:58.916288 (+ 9881us) write_op.cc:276] Timestamp: P: 1790537938903023 usec, L: 0
0927 19:38:58.919471 (+ 3183us) op_driver.cc:331] REPLICATION: starting
0927 19:38:59.077222 (+157751us) log.cc:842] Serialized 6449 byte log entry
0927 19:38:59.711255 (+634033us) op_driver.cc:444] REPLICATION: finished
0927 19:39:00.094548 (+383293us) write_op.cc:301] APPLY: starting
0927 19:39:00.094639 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:00.094922 (+ 283us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:00.095063 (+ 141us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:00.124877 (+ 29814us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:00.125155 (+ 278us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0927 19:39:00.125205 (+ 50us) write_op.cc:312] APPLY: finished
0927 19:39:00.125587 (+ 382us) log.cc:842] Serialized 113 byte log entry
0927 19:39:00.125760 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:00.125929 (+ 169us) write_op.cc:454] Released schema lock
0927 19:39:00.125981 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:00.141101 (+ 15120us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:00.141148 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3656,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":382931,"apply.run_cpu_time_us":4527,"apply.run_wall_time_us":47160,"num_ops":16,"prepare.queue_time_us":157057,"prepare.run_cpu_time_us":17423,"prepare.run_wall_time_us":359652,"raft.queue_time_us":354525,"raft.run_cpu_time_us":4768,"raft.run_wall_time_us":112549,"replication_time_us":791659,"thread_start_us":406341,"threads_started":4,"wal-append.queue_time_us":189898,"wal-append.run_cpu_time_us":19251,"wal-append.run_wall_time_us":1842015}]]}
I20260927 19:39:01.577454 2063 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:01.503197 4998 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:01.576220 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.574601 5007 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.574378 4915 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.589684 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.598340 2067 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:01.552317 4942 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:01.592291 2066 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:01.600129 4942 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.604158 5000 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:01.605284 1989 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:01.611893 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:01.616806 2139 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:01.629532 2139 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:01.636660 1962 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:01.640520 5001 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:39:01.642620 4887 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:01.643215 4887 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:01.644208 4887 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:01.646265 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.646873 2066 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.649020 2071 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:01.649256 2067 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260927 19:39:01.649664 2067 pending_rounds.cc:76] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 3
I20260927 19:39:01.649878 2067 pending_rounds.cc:97] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260927 19:39:01.649144 2686 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 5 to 7, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:01.650105 2067 raft_consensus.cc:2866] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:01.650377 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.651399 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.651707 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.652851 2064 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260927 19:39:01.653165 2064 pending_rounds.cc:76] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 4
I20260927 19:39:01.653386 2064 pending_rounds.cc:97] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260927 19:39:01.653595 2064 raft_consensus.cc:2866] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:01.650070 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.651404 2068 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:01.654577 2071 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:01.661163 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.662092 2066 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.611366 4998 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:01.611420 4942 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:01.665277 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.670297 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.679847 2069 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:01.685984 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.686761 2068 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:01.694960 2063 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:01.705971 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.706346 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.706949 2068 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:01.710956 2070 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:01.715242 2684 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 19:39:01.702251 2144 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:01.721217 2066 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.721756 2683 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 11 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260927 19:39:01.724720 1961 leader_election.cc:334] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.836s (SENT)
I20260927 19:39:01.727905 2063 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.729835 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.730643 2066 raft_consensus.cc:2354] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:39:01.739043 4963 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:39:01.743157 4791 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:01.743887 4791 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.746105 4962 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:01.749100 4791 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:01.750864 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:01.753903 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.754405 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.756072 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.756994 2071 raft_consensus.cc:2354] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:39:01.757392 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.758141 2069 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:01.758854 2071 raft_consensus.cc:3032] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:01.759305 2071 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:01.760408 2071 consensus_queue.cc:260] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.762820 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.763615 2069 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:01.763602 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.764523 2065 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39d534ed586444279970bb03de60c919"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:01.765070 2065 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:01.765410 2065 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:01.768766 2071 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:01.782083 2070 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:01.783298 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.784610 2063 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260927 19:39:01.782758 1961 leader_election.cc:334] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.887s (SENT)
I20260927 19:39:01.789175 2066 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:01.785553 2065 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.790802 2068 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260927 19:39:01.792871 1961 leader_election.cc:334] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.898s (SENT)
I20260927 19:39:01.796077 2071 raft_consensus.cc:1226] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:01.795058 2065 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:01.811969 2068 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:01.790995 5000 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:01.802347 5007 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260927 19:39:01.811519 4871 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:01.820340 4871 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.821579 5000 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.822683 4871 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:01.824059 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:01.824306 5000 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:01.825487 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:01.832321 2068 raft_consensus.cc:1226] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:01.833631 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.834604 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.835156 2069 raft_consensus.cc:2362] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5. Re-sending same reply.
I20260927 19:39:01.837653 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.837975 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.838251 2064 raft_consensus.cc:2362] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5. Re-sending same reply.
I20260927 19:39:01.838544 2066 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:01.839140 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.833930 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.840576 2070 raft_consensus.cc:2362] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7. Re-sending same reply.
I20260927 19:39:01.841480 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.833324 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.845911 2063 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:01.850908 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.851513 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.851611 2067 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:01.853050 2066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:39:01.853637 2066 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:01.853946 2066 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:39:01.858388 1961 leader_election.cc:334] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.940s (SENT)
I20260927 19:39:01.858839 1961 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:01.858032 2068 raft_consensus.cc:2362] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7. Re-sending same reply.
I20260927 19:39:01.861395 2063 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 19:39:01.859827 2067 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:01.863435 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.864297 2062 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:01.866708 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.869107 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.868377 5007 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:01.866761 2066 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:01.872331 2065 raft_consensus.cc:1261] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:39:01.872941 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.873730 2066 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:01.874244 2071 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:01.900244 2070 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:01.903736 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:01.904918 2063 raft_consensus.cc:2354] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 7. Current term is 8.
I20260927 19:39:01.906016 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:01.906958 2068 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:01.911554 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:01.932504 4461 consensus_queue.cc:1046] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:01.933727 4461 raft_consensus.cc:3032] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:01.934101 4461 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:01.934923 2144 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:39:01.934880 4461 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:01.935487 4963 consensus_queue.cc:1035] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:01.936468 4461 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:01.941766 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.942140 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.942750 1989 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:01.943033 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.943620 1993 raft_consensus.cc:1203] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-5.3] Dedup: 2.2->[5.3-5.3]
I20260927 19:39:01.943939 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.944267 5007 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:39:01.944779 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.943637 2145 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:01.944121 1993 raft_consensus.cc:1226] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [2.2-5.3]
I20260927 19:39:01.942901 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.948319 2139 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:01.971165 4625 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:01.980091 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.980232 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.980929 1987 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:01.980976 2139 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:01.984097 4625 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:01.986742 4962 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
W20260927 19:39:01.984979 1961 leader_election.cc:334] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 2.066s (SENT)
I20260927 19:39:01.984315 4887 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:01.988240 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.990839 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:01.990999 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:01.995359 1987 raft_consensus.cc:3032] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:01.995523 2139 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:01.995827 1987 raft_consensus.cc:729] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:01.996654 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.993041 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:01.993032 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:01.987180 4962 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:02.001118 1987 consensus_queue.cc:260] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.003288 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:02.005110 1961 leader_election.cc:398] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:39:02.005848 1987 raft_consensus.cc:3037] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.006954 1961 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.009197 1961 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:02.010579 2110 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:02.013406 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:02.014184 1994 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:02.013610 2139 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:02.020499 2110 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:02.028676 1987 raft_consensus.cc:2454] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:02.020815 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:02.032521 2145 raft_consensus.cc:3032] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:02.032832 2145 raft_consensus.cc:729] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:02.037772 2145 consensus_queue.cc:260] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.044855 2145 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:02.044448 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.014724 4461 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:02.029093 5021 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:02.052521 4461 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:02.053042 4461 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.015053 4995 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:02.071575 4995 consensus_queue.cc:1035] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:02.007850 5002 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260927 19:39:02.014847 4979 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260927 19:39:02.077685 4968 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:02.036298 4974 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:02.033849 4946 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.041640 4864 consensus_queue.cc:1046] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260927 19:39:02.056852 4791 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:02.021493 4625 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.059762 4962 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.039140 4998 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260927 19:39:02.108099 2145 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:02.098557 4915 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.061172 1994 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:02.052887 5021 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:02.117602 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 81765.1us
I20260927 19:39:02.089579 4946 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.120193 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
W20260927 19:39:02.123454 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 79108.7us
I20260927 19:39:02.142838 1987 raft_consensus.cc:595] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:39:02.143361 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 111729us
I20260927 19:39:02.143358 1987 raft_consensus.cc:603] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:02.143512 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.145303 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:02.146668 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.156128 4461 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.157037 4979 raft_consensus.cc:3032] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:02.157546 5055 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:39:02.159729 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:02.157583 4979 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:02.162405 4979 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.163235 5052 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:02.149036 4871 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:02.163743 5052 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:02.163898 4979 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.164171 5052 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.135500 4962 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:02.170089 5052 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.174306 5052 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:02.174520 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:02.178272 5021 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:02.179420 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:02.180373 4625 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.182297 4996 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.182790 4996 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.184782 4996 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:02.190308 4461 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:02.194706 4946 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:02.195217 4863 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:02.202785 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:02.203948 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 55202.9us
I20260927 19:39:02.204651 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.205088 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 157) took 3103 ms. Trace:
I20260927 19:39:02.209121 2679 consensus_queue.cc:237] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.209630 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:02.210187 1992 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:02.207513 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:02.206743 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:02.214884 1989 raft_consensus.cc:595] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:02.215233 1989 raft_consensus.cc:603] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:02.218994 5021 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:02.221925 2111 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.223225 5021 consensus_queue.cc:1046] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:39:02.223968 2143 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:02.224218 5000 raft_consensus.cc:3032] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:02.224623 5000 raft_consensus.cc:729] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.225432 4946 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:39:02.225577 5000 consensus_queue.cc:260] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.226678 5000 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:02.226987 2111 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:02.227768 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.208248 1893 rpcz_store.cc:276] 0927 19:38:59.101750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:59.102125 (+ 375us) service_pool.cc:227] Handling call
0927 19:38:59.209198 (+107073us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54002f58d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79198bcd6c 00007f79199b6c3e 00007f7917cc8fb4 00007f7917cc8f42 00007f7917cc8e71 00007f7917cc8dec 00007f7917cc80a2 00007f7911d8c94c 00007f7911d8bd86
0927 19:39:02.205010 (+2995812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:01.796282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:01.797082 (+ 800us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:01.797140 (+ 58us) write_op.cc:435] Acquired schema lock
0927 19:39:01.797168 (+ 28us) tablet.cc:660] Decoding operations
0927 19:39:01.798174 (+ 1006us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:01.798464 (+ 290us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:01.798515 (+ 51us) tablet.cc:683] Acquiring locks for 5 operations
0927 19:39:01.799350 (+ 835us) tablet.cc:699] Row locks acquired
0927 19:39:01.799381 (+ 31us) write_op.cc:260] PREPARE: finished
0927 19:39:01.799632 (+ 251us) write_op.cc:270] Start()
0927 19:39:01.799910 (+ 278us) write_op.cc:276] Timestamp: P: 1790537941799557 usec, L: 0
0927 19:39:01.799952 (+ 42us) op_driver.cc:331] REPLICATION: starting
0927 19:39:01.801302 (+ 1350us) log.cc:842] Serialized 2069 byte log entry
0927 19:39:01.925086 (+123784us) op_driver.cc:444] REPLICATION: finished
0927 19:39:01.932115 (+ 7029us) write_op.cc:301] APPLY: starting
0927 19:39:01.932210 (+ 95us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:01.994667 (+ 62457us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:01.998202 (+ 3535us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:02.061079 (+ 62877us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:02.061381 (+ 302us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0927 19:39:02.061456 (+ 75us) write_op.cc:312] APPLY: finished
0927 19:39:02.061742 (+ 286us) log.cc:842] Serialized 47 byte log entry
0927 19:39:02.062010 (+ 268us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:02.062151 (+ 141us) write_op.cc:454] Released schema lock
0927 19:39:02.062215 (+ 64us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:02.062559 (+ 344us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:02.062603 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":446905,"reported_tablets":24,"spinlock_wait_cycles":99024256,"child_traces":[["op",{"apply.queue_time_us":6723,"apply.run_cpu_time_us":3771,"apply.run_wall_time_us":130918,"num_ops":5,"prepare.queue_time_us":180403,"prepare.run_cpu_time_us":17852,"prepare.run_wall_time_us":132429,"raft.queue_time_us":13960,"raft.run_cpu_time_us":906,"raft.run_wall_time_us":7704,"replication_time_us":125028,"thread_start_us":221480,"threads_started":4,"wal-append.queue_time_us":99916}]]}
I20260927 19:39:02.228942 4920 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:02.230412 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:02.233695 4920 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:02.233850 1989 raft_consensus.cc:595] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:02.234216 4920 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:02.236238 4974 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.237342 2683 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEARNER]: Advancing to term 8
I20260927 19:39:02.237206 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.238551 5021 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.239254 5021 consensus_queue.cc:1046] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:39:02.240141 4946 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:02.240484 5057 raft_consensus.cc:3032] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:02.273841 5057 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.239552 2143 raft_consensus.cc:1226] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:02.251343 2683 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:02.257575 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.233744 4887 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.025s
I20260927 19:39:02.260547 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:02.377817 1987 raft_consensus.cc:595] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:02.377928 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 120442us
I20260927 19:39:02.378741 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.379464 4871 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:02.380010 4871 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.380133 4791 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 19:39:02.380605 2037 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.381497 4942 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.382272 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.383342 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.384048 5055 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.385043 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:02.385787 5055 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:02.385722 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c802eef7b2904ec984925e3999cff361"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:02.386300 5055 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:02.386679 5055 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.375905 4461 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.380963 4871 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.387389 4461 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.389430 4461 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:02.391434 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:02.386711 2143 raft_consensus.cc:595] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:02.392071 1987 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:02.392191 2143 raft_consensus.cc:603] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:02.393854 2111 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.394279 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.395447 4942 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:02.395633 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:39:02.395812 4942 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:02.396169 4942 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.394599 4461 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:02.397233 4461 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:02.397614 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:02.398289 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:02.397616 4461 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:02.407512 4942 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.412515 2139 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:02.413731 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.414620 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.415285 4791 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:02.415690 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.416213 4962 raft_consensus.cc:979] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:39:02.417631 4962 raft_consensus.cc:1067] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:39:02.417928 5055 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.418938 4946 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.419658 4946 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.420011 5055 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:02.377899 5021 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:39:02.420614 4864 consensus_queue.cc:1046] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:39:02.421725 4942 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:02.421868 4887 raft_consensus.cc:3032] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:02.422325 4887 raft_consensus.cc:729] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:02.423187 4887 consensus_queue.cc:260] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.425248 4887 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.429651 4461 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.430760 5057 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.431890 5057 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:02.432308 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:02.433053 1987 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:02.434150 2037 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.435001 2037 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.439154 5000 raft_consensus.cc:2781] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:02.439203 4871 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:39:02.439764 5000 raft_consensus.cc:686] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.440023 4871 raft_consensus.cc:686] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.440662 5000 consensus_queue.cc:237] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.441169 4871 consensus_queue.cc:237] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.448696 4461 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:02.450407 5052 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.450896 5052 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.452840 5052 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:02.455677 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:02.458635 5064 consensus_queue.cc:1046] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:39:02.459052 2143 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:39:02.459648 2143 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:02.460011 2143 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:02.465718 1994 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:02.467381 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:02.467739 4963 raft_consensus.cc:3032] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:02.468068 4963 raft_consensus.cc:729] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:02.468834 4963 consensus_queue.cc:260] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.469844 4963 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.259397 1892 catalog_manager.cc:5495] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:02.481475 5001 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:02.492082 1994 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:39:02.481235 1892 catalog_manager.cc:5495] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 5 to 6, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 6 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:02.498597 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:02.499063 1989 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:02.501084 1987 raft_consensus.cc:2454] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
I20260927 19:39:02.503608 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:02.504210 1987 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.509205 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:02.510555 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:02.512101 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:02.512187 1987 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:02.514258 2062 raft_consensus.cc:3032] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:02.514537 2140 raft_consensus.cc:1226] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:02.514631 2062 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.515661 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:02.507970 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:02.511473 2064 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:02.497049 1892 catalog_manager.cc:5495] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:02.510907 2143 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.518150 2143 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:02.517517 1892 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 5, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:02.515486 2111 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.520732 2137 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:02.521026 1892 catalog_manager.cc:5495] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:02.522527 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:02.523092 2140 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.524134 2137 raft_consensus.cc:1203] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-5.4] Dedup: 5.4->[]
I20260927 19:39:02.524621 2137 raft_consensus.cc:1226] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260927 19:39:02.518954 2062 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.512770 2063 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:39:02.526613 2110 leader_election.cc:398] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 7. Current term is 8.
I20260927 19:39:02.526921 2110 leader_election.cc:401] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260927 19:39:02.525986 2062 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:02.431759 4920 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.525959 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:02.528502 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:02.528990 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:02.529681 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:02.532974 1989 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:02.533308 2679 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:02.533869 2143 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.533327 1989 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:02.537948 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:02.538563 2065 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.539601 2069 pending_rounds.cc:76] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 1
I20260927 19:39:02.539904 2069 pending_rounds.cc:97] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260927 19:39:02.540122 2069 raft_consensus.cc:2866] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:02.540400 2069 pending_rounds.cc:97] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260927 19:39:02.540608 2069 raft_consensus.cc:2866] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:02.538076 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:02.525601 4864 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.020s
I20260927 19:39:02.535907 1989 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.546311 1989 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:02.571403 1989 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:02.572775 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.573599 2062 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:39:02.574182 2143 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:02.574997 2140 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:02.540630 4996 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 7. Current term is 8.
I20260927 19:39:02.537731 4995 raft_consensus.cc:2781] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:02.593238 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:02.591951 2065 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:02.598019 2071 raft_consensus.cc:1226] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: [6.6-6.7]
I20260927 19:39:02.604624 2065 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:02.604683 2679 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:02.613134 2065 raft_consensus.cc:1261] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260927 19:39:02.614886 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 80028us
W20260927 19:39:02.625104 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 52460.8us
I20260927 19:39:02.605903 4871 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:02.610280 4996 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:02.607795 4995 raft_consensus.cc:686] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:02.593585 5021 consensus_queue.cc:1046] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260927 19:39:02.638208 2110 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:02.646874 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:02.608409 4946 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260927 19:39:02.661144 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:02.662062 1989 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 6
I20260927 19:39:02.671563 1989 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:02.673739 2111 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:39:02.672248 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 71877.1us
I20260927 19:39:02.682513 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:02.657034 4996 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:02.661067 5052 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:02.665635 5074 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:02.701156 5069 raft_consensus.cc:2781] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:02.699723 4920 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:02.731968 2140 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:39:02.734258 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:02.700524 5074 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:02.723335 4625 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:02.718578 4946 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:02.738559 4920 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.719627 4887 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.030s
I20260927 19:39:02.720109 4968 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.029s
I20260927 19:39:02.741794 5062 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:02.737637 5074 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.741963 5069 raft_consensus.cc:686] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:02.765117 4887 consensus_queue.cc:1225] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer 4cc2df048f424e08803a390946004df0 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260927 19:39:02.755365 4946 raft_consensus.cc:686] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.746459 4995 consensus_queue.cc:237] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.752012 4625 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.807482 4864 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.817788 4864 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.818910 4625 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:02.756830 5075 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
W20260927 19:39:02.815024 5073 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260927 19:39:02.819901 4864 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:02.822052 5075 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:02.822588 5075 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.691771 5054 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:39:02.824684 5075 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:02.824798 4871 raft_consensus.cc:3032] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:02.750937 4996 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:02.759442 4920 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.861330 5062 raft_consensus.cc:686] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.868036 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:02.825208 4871 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:02.879415 4871 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.880679 4871 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:02.873636 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:02.876686 2070 raft_consensus.cc:2379] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:02.929575 2141 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
W20260927 19:39:02.930125 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 122232us
I20260927 19:39:02.936899 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:02.937911 2683 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:02.955665 1959 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:39:02.957227 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 194934us
I20260927 19:39:02.961952 4864 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:02.962586 4864 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.963351 4979 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:02.963804 4979 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:02.964143 4979 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:02.964808 4864 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:02.942734 4946 consensus_queue.cc:237] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:02.969182 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:02.969967 2070 raft_consensus.cc:3037] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:02.972764 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:02.973812 2138 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:02.975358 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:02.975883 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:02.976066 1987 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:02.976576 2143 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:02.978425 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:02.981138 2065 raft_consensus.cc:2379] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:02.990650 2067 raft_consensus.cc:1203] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-6.5] Dedup: 3.3->[4.4-6.5]
I20260927 19:39:02.991276 2067 pending_rounds.cc:76] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 3
I20260927 19:39:02.991583 2067 pending_rounds.cc:97] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260927 19:39:02.991885 2067 raft_consensus.cc:2866] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:02.993512 2070 raft_consensus.cc:2454] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
W20260927 19:39:03.033454 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 77922.1us
I20260927 19:39:03.034778 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:03.043085 2138 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:03.046466 4920 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:03.048513 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:03.049204 1987 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:03.050264 5096 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:03.054296 2138 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:39:03.057906 1993 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:39:03.075446 2111 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:02.986238 5069 consensus_queue.cc:237] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.090229 1987 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9. Ops: []
I20260927 19:39:03.093751 1988 raft_consensus.cc:1261] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:03.091874 5088 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:03.053658 5086 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260927 19:39:03.123834 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 48524.5us
I20260927 19:39:03.073009 5071 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.077142 4920 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.137778 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:03.133296 5086 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:39:03.231352 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 222691us
I20260927 19:39:03.100090 4979 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.112571 5094 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260927 19:39:03.131604 5088 consensus_queue.cc:1035] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
W20260927 19:39:03.321182 5085 log.cc:924] Time spent T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
I20260927 19:39:03.143659 5071 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.143919 4920 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:03.480177 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 303373us
W20260927 19:39:03.541559 5103 log.cc:924] Time spent T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.268s user 0.003s sys 0.006s
I20260927 19:39:03.541728 2062 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:39:03.544219 5069 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:03.579216 5069 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.581913 5069 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:03.131716 4461 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:03.031502 5062 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.104660 5096 consensus_queue.cc:1046] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260927 19:39:03.583766 4461 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:03.584239 4461 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:03.584378 5096 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:03.585157 5096 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.553341 5099 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:39:03.556751 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 378050us
I20260927 19:39:03.588155 5096 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:03.586442 5099 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.591878 5099 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:03.564684 5089 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:03.592489 4461 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.593019 5089 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.594627 4461 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:03.555064 5086 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:03.614790 5105 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:03.615051 4979 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:03.616169 5088 raft_consensus.cc:3032] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:03.618566 4920 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:03.567412 5104 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.625752 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:03.594983 5089 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:03.618955 5066 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 1.004s user 0.004s sys 0.000s
I20260927 19:39:03.570608 5109 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.634528 2069 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2220) took 1111 ms. Trace:
W20260927 19:39:03.635481 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 82266us
I20260927 19:39:03.610149 5100 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.661010 1962 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:39:03.635206 5088 raft_consensus.cc:729] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:03.658346 2069 rpcz_store.cc:276] 0927 19:39:02.510815 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.511861 (+ 1046us) service_pool.cc:227] Handling call
0927 19:39:02.539437 (+ 27576us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:02.541117 (+ 1680us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:02.541145 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:02.616146 (+ 75001us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b2c00000238. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:39:02.616622 (+ 476us) log.cc:842] Serialized 26 byte log entry
0927 19:39:02.801310 (+184688us) raft_consensus.cc:1631] Marking committed up to 2
0927 19:39:02.976601 (+175291us) log.cc:842] Serialized 14 byte log entry
0927 19:39:02.979651 (+ 3050us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:02.979768 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:03.609995 (+630227us) raft_consensus.cc:1671] finished
0927 19:39:03.621787 (+ 11792us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:03.621953 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36227,"prepare.queue_time_us":80560,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":4278,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":21343,"spinlock_wait_cycles":9722624,"thread_start_us":255715,"threads_started":2,"wal-append.queue_time_us":423703}
I20260927 19:39:03.636149 4461 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.717592 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:03.636672 5105 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.713474 5071 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:03.711385 5119 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:39:03.658269 5104 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.774238 5088 consensus_queue.cc:260] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.793622 5121 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.794338 5121 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.796972 5121 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:03.799005 5078 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 1.010s user 0.003s sys 0.003s
I20260927 19:39:03.800154 5118 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:39:03.803545 5112 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260927 19:39:03.810050 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:03.852160 2686 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:03.875763 2686 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:03.782874 5115 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:03.784018 4791 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.663172 5109 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.883311 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:03.787173 5099 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:03.786332 5119 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:03.665172 5100 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.863541 5088 raft_consensus.cc:3037] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:03.723187 4461 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.906538 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:03.911800 1996 raft_consensus.cc:3037] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:03.864936 5121 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.926795 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:03.928726 2137 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:39:03.927088 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:03.928601 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:39:03.901304 5074 consensus_meta.cc:287] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 1.112s user 0.012s sys 0.007s
I20260927 19:39:03.899925 5120 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:03.945621 1990 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260927 19:39:03.911909 5054 consensus_meta.cc:287] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 1.087s user 0.009s sys 0.011s
I20260927 19:39:03.915917 4979 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:03.957187 1994 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 541) took 1458 ms. Trace:
I20260927 19:39:03.866636 5118 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.908533 5091 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:03.907857 5119 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:03.961035 1996 raft_consensus.cc:2454] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:03.960412 1994 rpcz_store.cc:276] 0927 19:39:02.498801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.499849 (+ 1048us) service_pool.cc:227] Handling call
0927 19:39:02.500493 (+ 644us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:02.500880 (+ 387us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:02.500916 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:02.546084 (+ 45168us) log.cc:842] Serialized 501 byte log entry
0927 19:39:02.694798 (+148714us) raft_consensus.cc:1631] Marking committed up to 7
0927 19:39:02.694836 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:02.694951 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:03.798363 (+1103412us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:03.956889 (+158526us) raft_consensus.cc:1671] finished
0927 19:39:03.956909 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:03.957131 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3075840,"thread_start_us":111435,"threads_started":1,"wal-append.queue_time_us":169900}
I20260927 19:39:03.907424 4791 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:03.983858 1995 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:03.982340 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:03.910697 5099 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.955262 5071 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:03.934526 5121 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.984711 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:03.955567 5123 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.987417 2138 raft_consensus.cc:3037] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.000607 4461 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.004920 4461 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.005548 4461 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.010207 1992 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:04.010614 2062 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2232) took 1051 ms. Trace:
I20260927 19:39:03.948339 5074 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.987452 5122 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.981385 5135 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.018630 4863 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:03.911094 5115 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.011188 2062 rpcz_store.cc:276] 0927 19:39:02.959286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.959525 (+ 239us) service_pool.cc:227] Handling call
0927 19:39:02.960373 (+ 848us) raft_consensus.cc:1457] Updating replica for 3 ops
0927 19:39:02.980533 (+ 20160us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:02.980560 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0927 19:39:02.981633 (+ 1073us) log.cc:842] Serialized 558 byte log entry
0927 19:39:03.541590 (+559957us) raft_consensus.cc:1631] Marking committed up to 8
0927 19:39:03.621145 (+ 79555us) log.cc:842] Serialized 14 byte log entry
0927 19:39:03.802626 (+181481us) spinlock_profiling.cc:243] Waited 64.6 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:03.908565 (+105939us) log.cc:842] Serialized 14 byte log entry
0927 19:39:03.914382 (+ 5817us) log.cc:842] Serialized 14 byte log entry
0927 19:39:03.914759 (+ 377us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:03.928240 (+ 13481us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.010352 (+ 82112us) raft_consensus.cc:1671] finished
0927 19:39:04.010371 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.010542 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15769,"prepare.queue_time_us":419268,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":21604,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":34292,"spinlock_wait_cycles":170876032,"thread_start_us":726343,"threads_started":2,"wal-append.queue_time_us":624643}
I20260927 19:39:03.979665 5113 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.994153 4871 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:03.986094 5116 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:04.007397 5130 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.030200 2140 raft_consensus.cc:2354] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:39:03.976414 4979 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.029603 4461 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.007367 5133 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:04.049672 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:03.975996 5091 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:03.959712 5120 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.001739 5071 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:04.082350 2037 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.008s sys 0.025s
W20260927 19:39:04.082890 2037 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.534s user 0.008s sys 0.025s
I20260927 19:39:04.048604 5062 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.089215 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:39:04.084338 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 58304.6us
I20260927 19:39:04.029240 4863 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.031778 4871 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.092578 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.094657 2144 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:04.031965 5116 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.017876 5105 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.048306 5104 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.002192 5123 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.025574 5122 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.033507 5113 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.094596 5100 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.102401 4791 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.028680 5135 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.089984 5062 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.101883 5143 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.052776 5133 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.047595 5130 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.136094 5071 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.137791 5115 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.139416 5116 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:39:04.140736 5083 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 1.208s user 0.010s sys 0.001s
I20260927 19:39:04.142529 4979 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.143595 5074 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.145169 5122 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.131940 4791 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.171741 5120 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.172042 5130 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.177122 2138 raft_consensus.cc:2454] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
W20260927 19:39:04.186002 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 272687us
I20260927 19:39:04.146435 4863 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.166776 5074 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:39:04.189226 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 265558us
I20260927 19:39:04.146476 5123 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:04.189775 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 207397us
I20260927 19:39:04.168203 5121 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:04.200717 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 112130us
I20260927 19:39:04.201934 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.203014 2111 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.203064 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.206323 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.206945 1993 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:39:04.207933 2679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34936 (request call id 897) took 1518 ms. Trace:
I20260927 19:39:04.204461 1996 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 3.
I20260927 19:39:04.208313 2679 rpcz_store.cc:276] 0927 19:39:02.685508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.685664 (+ 156us) service_pool.cc:227] Handling call
0927 19:39:02.686099 (+ 435us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:39:02.686572 (+ 473us) raft_consensus.cc:1520] Early marking committed up to index 5
0927 19:39:02.686614 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:39:02.687248 (+ 634us) log.cc:842] Serialized 52 byte log entry
0927 19:39:02.744237 (+ 56989us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:39:02.744297 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:02.744394 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.140188 (+1395794us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:04.195757 (+ 55569us) raft_consensus.cc:1671] finished
0927 19:39:04.195787 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.203311 (+ 7524us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1224,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":2752,"spinlock_wait_cycles":53976832,"thread_start_us":56555,"threads_started":1,"wal-append.queue_time_us":181572}
I20260927 19:39:04.185004 5118 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.188797 5120 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:04.216245 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.217307 5137 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260927 19:39:04.217904 5133 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.218117 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.218688 5148 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.218758 1996 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:04.219317 5148 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.220228 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.220884 1996 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.211738 5141 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.222059 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:04.222681 1996 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 7
I20260927 19:39:04.222275 5141 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.192772 5135 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.226604 1993 raft_consensus.cc:1261] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260927 19:39:04.185714 4791 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:04.214087 5086 consensus_meta.cc:287] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.580s user 0.010s sys 0.001s
I20260927 19:39:04.134857 5143 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.202363 5074 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.226553 5148 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.212499 5123 raft_consensus.cc:2781] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:04.230245 5154 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:04.238903 5154 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.239646 4791 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.240295 5150 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.240587 5148 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.240872 5150 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.241389 5154 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.241134 5148 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.243046 5150 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.243427 5148 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.231042 5091 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.232201 5140 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.245579 5140 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.245618 5154 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Committing config change with OpId 8.8: config changed from index 4 to 8, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:04.247022 5113 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.247911 5140 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.248400 5062 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.243937 5099 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.243916 5148 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.255748 2141 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 1333) took 1427 ms. Trace:
I20260927 19:39:04.256294 2141 rpcz_store.cc:276] 0927 19:39:02.828708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.829386 (+ 678us) service_pool.cc:227] Handling call
0927 19:39:04.255538 (+1426152us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b5400076cd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:39:04.255662 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13871360}
W20260927 19:39:04.266233 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 77076.9us
I20260927 19:39:04.267643 1959 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:04.268446 4887 raft_consensus.cc:2781] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:04.226750 5120 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.232342 5143 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.232928 5074 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.255784 5148 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.271804 5120 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.230827 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.274623 4887 raft_consensus.cc:686] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:04.275315 5109 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.275632 4887 consensus_queue.cc:237] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.273198 1993 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.280588 5000 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.281133 5000 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.283876 5000 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.283900 5141 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.288530 5150 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.289075 5150 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.291844 5150 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.293741 5086 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.296732 5086 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.300282 5141 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.301481 5141 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.303555 5141 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.309881 2066 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:04.311986 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48570 (request call id 1767) took 1175 ms. Trace:
I20260927 19:39:04.312454 2066 rpcz_store.cc:276] 0927 19:39:03.136940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:03.137510 (+ 570us) service_pool.cc:227] Handling call
0927 19:39:04.311699 (+1174189us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400323f54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:39:04.311888 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2693298048}
W20260927 19:39:04.316493 5079 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 1.330s user 0.000s sys 0.018s
I20260927 19:39:04.319739 5118 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.320258 5118 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.321700 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.322228 5118 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.324105 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.324810 2684 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 3.
I20260927 19:39:04.325723 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.327106 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.327948 2680 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:04.273150 5148 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.329311 5148 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.329840 5148 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.331933 5148 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.230778 4871 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.334199 2654 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:04.335179 5092 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:39:04.335562 5092 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:04.335934 5092 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
W20260927 19:39:04.327118 2653 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.730s user 0.010s sys 0.009s
W20260927 19:39:04.337000 2653 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.740s user 0.010s sys 0.009s
W20260927 19:39:04.337260 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 1.74064e+06us
I20260927 19:39:04.341615 2683 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:04.344187 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.358062 2067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2234) took 1368 ms. Trace:
I20260927 19:39:04.360882 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.362196 1996 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.345110 2682 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:04.358639 2067 rpcz_store.cc:276] 0927 19:39:02.989739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.989930 (+ 191us) service_pool.cc:227] Handling call
0927 19:39:02.990419 (+ 489us) raft_consensus.cc:1457] Updating replica for 3 ops
0927 19:39:02.992622 (+ 2203us) raft_consensus.cc:1520] Early marking committed up to index 3
0927 19:39:02.993250 (+ 628us) log.cc:842] Serialized 14 byte log entry
0927 19:39:04.307268 (+1314018us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b180035eae0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f79135f182c 00007f7910a1c9dc 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e957b
0927 19:39:04.307399 (+ 131us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:39:04.308083 (+ 684us) log.cc:842] Serialized 52 byte log entry
0927 19:39:04.308258 (+ 175us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:39:04.308515 (+ 257us) log.cc:842] Serialized 14 byte log entry
0927 19:39:04.328829 (+ 20314us) log.cc:842] Serialized 14 byte log entry
0927 19:39:04.329053 (+ 224us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:04.329211 (+ 158us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.357768 (+ 28557us) raft_consensus.cc:1671] finished
0927 19:39:04.357797 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.357990 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2499,"prepare.queue_time_us":20150,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":87,"raft.queue_time_us":46795,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":11230,"spinlock_wait_cycles":6408448,"thread_start_us":40132,"threads_started":1,"wal-append.queue_time_us":138862}
I20260927 19:39:04.369313 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.370530 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.371304 2679 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:04.376312 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.377343 1962 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.237124 5056 raft_consensus.cc:979] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:04.387936 1962 leader_election.cc:398] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:39:04.369958 1996 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 4 to 8, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:04.388267 1962 leader_election.cc:401] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:04.385186 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.392256 2685 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:39:04.393946 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.244414 5119 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.373265 2683 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEARNER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:39:04.402361 1987 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:04.409190 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:39:04.391745 5079 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.069s user 0.000s sys 0.003s
W20260927 19:39:04.400233 5107 log.cc:924] Time spent T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 1.202s user 0.007s sys 0.013s
I20260927 19:39:04.419404 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:39:04.434188 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 69354.3us
I20260927 19:39:04.424625 1959 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:04.437952 1959 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:04.434429 4461 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:04.447199 2108 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.447860 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.448019 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.448524 2136 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:04.448987 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.449452 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.449724 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.450487 2144 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:04.450870 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.451838 2136 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:04.387547 5063 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:04.406574 4968 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.454694 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.458827 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.503943 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:04.458792 2140 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:39:04.504813 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 43607.8us
I20260927 19:39:04.458009 2138 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 5: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:04.412058 5121 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:04.523625 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.518357 2064 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2222) took 2006 ms. Trace:
I20260927 19:39:04.418180 4962 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:04.434119 4863 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7.
I20260927 19:39:04.524055 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.566413 2067 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:04.567263 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.567847 2067 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:04.434569 5123 raft_consensus.cc:686] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:04.520939 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.571403 2068 raft_consensus.cc:2454] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:04.572201 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.572923 2068 raft_consensus.cc:2379] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 7: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:04.520447 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:04.521145 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.526512 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.521625 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.518429 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.414279 5124 consensus_queue.cc:1035] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.125s
I20260927 19:39:04.440542 5153 raft_consensus.cc:1067] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:04.556955 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.592182 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.554446 5148 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.530728 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.530243 1962 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.564812 2071 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.568583 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.443363 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.570691 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.575067 2062 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:04.599804 2069 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:39:04.574893 2065 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.521347 2137 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:04.601156 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.576678 1989 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1338) took 1846 ms. Trace:
I20260927 19:39:04.502321 5096 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.578840 2066 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:04.573638 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.608523 2068 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:39:04.608788 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.609275 2066 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:04.601794 2065 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:04.501804 5126 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.578938 2111 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:04.613034 2111 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.525622 4968 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.586436 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.617997 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.587193 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.589687 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:04.618929 2070 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:04.523415 4871 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:39:04.570056 5078 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260927 19:39:04.542052 4461 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.635476 4461 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:04.576552 5083 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260927 19:39:04.565688 5113 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:04.516443 5147 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260927 19:39:04.600948 5069 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:04.563896 2064 rpcz_store.cc:276] 0927 19:39:02.512437 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.516890 (+ 4453us) service_pool.cc:227] Handling call
0927 19:39:02.517136 (+ 246us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:02.530932 (+ 13796us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:02.530962 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:02.629923 (+ 98961us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b2c00000238. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:39:02.630421 (+ 498us) log.cc:842] Serialized 26 byte log entry
0927 19:39:02.790176 (+159755us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:39:02.811126 (+ 20950us) log.cc:842] Serialized 14 byte log entry
0927 19:39:02.861543 (+ 50417us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:02.861616 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.315660 (+1454044us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:04.518158 (+202498us) raft_consensus.cc:1671] finished
0927 19:39:04.518181 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.518301 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2889,"prepare.queue_time_us":91500,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":68175,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":15122,"spinlock_wait_cycles":55685120,"thread_start_us":197482,"threads_started":2,"wal-append.queue_time_us":298956}
I20260927 19:39:04.653422 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.654278 2064 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:04.655344 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.602205 1989 rpcz_store.cc:276] 0927 19:39:02.729848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.729986 (+ 138us) service_pool.cc:227] Handling call
0927 19:39:02.733190 (+ 3204us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:02.733527 (+ 337us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:02.733553 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:02.733922 (+ 369us) log.cc:842] Serialized 26 byte log entry
0927 19:39:02.941955 (+208033us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b18003720a0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a1c345 00007f7919853a39 00007f7910a398bb 00007f7910a37cbf 00007f7910a39e22 00007f79135e35bb 00007f79135e9063 00007f79135e9411 00007f79139365cf
0927 19:39:03.087548 (+145593us) raft_consensus.cc:1631] Marking committed up to 4
0927 19:39:03.087604 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:03.087715 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.269151 (+1181436us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b2400195638. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910892cf4 00007f7910890dfb 00007f791091f559 00007f791091f1e5 00007f791091e82d 00007f79135cd46e 00007f79135e4c0c 00007f79135e3f36 00007f79135e3b58 00007f79135ed551
0927 19:39:04.576361 (+307210us) raft_consensus.cc:1671] finished
0927 19:39:04.576402 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.576603 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2109,"raft.queue_time_us":116,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9885,"spinlock_wait_cycles":17354880,"thread_start_us":145369,"threads_started":1,"wal-append.queue_time_us":402002}
I20260927 19:39:04.604168 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.604786 5138 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.617873 5088 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.605147 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.515074 5063 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.602334 5021 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:39:04.605630 5117 log.cc:924] Time spent T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.704s user 0.001s sys 0.003s
I20260927 19:39:04.609325 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.656071 2064 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:04.609555 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.522765 5105 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:04.670087 2067 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:04.610363 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.610456 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.610396 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:04.675164 2062 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:04.623493 2111 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.617830 5096 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:04.637218 2136 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:39:04.673864 2064 raft_consensus.cc:1261] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260927 19:39:04.571934 4962 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.642416 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:04.598155 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 67982.7us
I20260927 19:39:04.602654 4863 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:04.650046 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.673388 4461 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:04.662287 5165 log.cc:924] Time spent T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260927 19:39:04.602023 5148 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.646936 5113 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.637821 5137 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.633220 5109 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:04.668697 2070 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.672359 2068 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:04.570415 5121 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.674868 2065 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:04.605916 5158 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:04.674422 5105 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.679164 4461 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.631594 4968 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.640633 5126 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.651090 5147 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:04.697379 2063 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48556 (request call id 2224) took 2184 ms. Trace:
I20260927 19:39:04.726590 2682 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 7
I20260927 19:39:04.698995 4962 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.699499 1990 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:04.705484 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.706571 5101 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:04.670996 5119 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.710884 5142 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:39:04.711203 5164 log.cc:924] Time spent T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260927 19:39:04.674515 5021 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.711426 5125 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.711884 5160 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.737792 5160 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.666615 5123 consensus_queue.cc:237] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.740723 4968 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.742456 5142 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.742869 2142 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Committing config change with OpId 5.6: config changed from index -1 to 6, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:39:04.743969 4968 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.680047 4863 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.641670 4871 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:04.678517 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 55165.7us
I20260927 19:39:04.718556 4461 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.675484 5088 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.719828 1988 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60296 (request call id 1345) took 1587 ms. Trace:
W20260927 19:39:04.720780 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 190865us
I20260927 19:39:04.701941 5113 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:04.753698 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.754756 2686 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.701215 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.727659 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:04.708060 1962 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.758966 1995 raft_consensus.cc:595] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:04.759140 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 51155us
I20260927 19:39:04.723668 2063 rpcz_store.cc:276] 0927 19:39:02.512924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.537762 (+ 24838us) service_pool.cc:227] Handling call
0927 19:39:02.538075 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:02.539134 (+ 1059us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:02.539159 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:02.759716 (+220557us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b1800334660. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79198d98b1 00007f79198e6661 00007f7910a07f96 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a
0927 19:39:03.140192 (+380476us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b4800271158. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79188097f1 00007f79188008e7 00007f791880077a 00007f79188f495f 00007f79188f4627 00007f7918908b2c 00007f7918908ad8 00007f7918908a58 00007f7918908a13 00007f7918907cbc 00007f7917889be6
0927 19:39:04.337768 (+1197576us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b2c00000238. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:39:04.338218 (+ 450us) log.cc:842] Serialized 26 byte log entry
0927 19:39:04.505790 (+167572us) raft_consensus.cc:1631] Marking committed up to 5
0927 19:39:04.552724 (+ 46934us) log.cc:842] Serialized 14 byte log entry
0927 19:39:04.583440 (+ 30716us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:04.583537 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.697084 (+113547us) raft_consensus.cc:1671] finished
0927 19:39:04.697112 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.697285 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49745,"prepare.queue_time_us":46620,"prepare.run_wall_time_us":59,"raft.queue_time_us":44959,"raft.run_cpu_time_us":4487,"raft.run_wall_time_us":594213,"spinlock_wait_cycles":123246976,"thread_start_us":167214,"threads_started":1,"wal-append.queue_time_us":267751}
I20260927 19:39:04.671334 5069 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.774173 2143 raft_consensus.cc:1226] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9. Ops: []
I20260927 19:39:04.774533 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.775631 5054 raft_consensus.cc:1050] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: attempting to promote NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 to VOTER
I20260927 19:39:04.713052 5121 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.731379 5101 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.778641 5101 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:04.778857 5153 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.732306 5096 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.779414 5153 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.782842 5092 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.784065 2679 raft_consensus.cc:1226] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:39:04.785640 5069 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.788043 4871 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.789815 5148 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.721055 5105 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:04.735502 2140 raft_consensus.cc:2354] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:39:04.737254 5021 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.741678 5125 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.674858 5138 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:04.712901 5166 log.cc:924] Time spent T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260927 19:39:04.748950 4461 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.749782 2111 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.705572 5137 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.756124 2654 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.750499 1988 rpcz_store.cc:276] 0927 19:39:03.132424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:03.132563 (+ 139us) service_pool.cc:227] Handling call
0927 19:39:03.132926 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:03.137684 (+ 4758us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:03.137727 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:03.175972 (+ 38245us) log.cc:842] Serialized 26 byte log entry
0927 19:39:03.552220 (+376248us) raft_consensus.cc:1631] Marking committed up to 3
0927 19:39:03.552262 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:03.552330 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:04.300587 (+748257us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:04.700770 (+400183us) raft_consensus.cc:1671] finished
0927 19:39:04.700794 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:04.719742 (+ 18948us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4352,"raft.queue_time_us":9731,"raft.run_cpu_time_us":8151,"raft.run_wall_time_us":379432,"spinlock_wait_cycles":51559680,"thread_start_us":10392,"threads_started":1,"wal-append.queue_time_us":364431}
I20260927 19:39:04.727739 5147 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:04.762513 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:04.807542 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 51473.8us
I20260927 19:39:04.748471 4863 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:04.755375 5074 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260927 19:39:04.813300 5126 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.814849 5063 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.767160 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:04.768842 5088 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:04.770262 2139 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40150 (request call id 1326) took 2256 ms (client timeout 2359 ms). Trace:
I20260927 19:39:04.815828 2141 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 8
W20260927 19:39:04.815954 2139 rpcz_store.cc:269] 0927 19:39:02.514024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.516624 (+ 2600us) service_pool.cc:227] Handling call
0927 19:39:04.770045 (+2253421us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b5400263c54. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79139662ae 00007f79188a3c27 00007f7911f8bc04 00007f7911f8bb92 00007f7911f8bac1 00007f7911f8ba3c 00007f7911f8acf2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:39:04.770171 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5119382272}
I20260927 19:39:04.816993 5125 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.818858 5138 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:04.821554 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 71877.8us
I20260927 19:39:04.771353 4968 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.777940 2682 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260927 19:39:04.797848 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:04.827476 1995 raft_consensus.cc:595] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:04.713581 5158 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.764243 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.789094 5113 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.825134 2141 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:04.822247 2111 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.828433 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.828914 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.836252 2685 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:04.806427 4920 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.831373 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.833150 2108 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.834372 2144 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:04.810426 5137 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.837993 2111 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.840497 1962 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.820708 5128 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:04.845929 2108 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.806157 5160 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.850780 4461 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.829902 4968 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.851572 1962 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.852783 5081 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.852806 1989 raft_consensus.cc:1226] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 9. Current term is 10. Ops: []
I20260927 19:39:04.854022 2137 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:04.857793 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:46928 (request call id 7) took 15659 ms. Trace:
I20260927 19:39:04.837211 5126 raft_consensus.cc:2741] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260927 19:39:04.835551 5153 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:04.853658 4968 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.889183 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.859251 2111 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.847070 5074 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.861970 4461 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:04.854671 5121 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:04.862171 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:04.903551 2685 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.846838 4920 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.862651 5093 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:39:04.858909 5109 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.855604 5081 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.860267 2142 rpcz_store.cc:276] 0927 19:38:49.196077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:49.231248 (+ 35171us) service_pool.cc:227] Handling call
0927 19:38:49.872632 (+641384us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b58007a09b0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138ddf63 00007f7913941aba 00007f791395da7a 00007f791395eada 00007f791395dc8b 00007f7913969084 00007f7913968743 00007f79188a4f39 00007f7911f90704 00007f7911f90692
0927 19:38:50.766976 (+894344us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b58004445b0. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f79138ddf63 00007f7913941aba 00007f791395da7a 00007f791395eada 00007f791395dc8b 00007f7913969084 00007f7913968743 00007f79188a4f39 00007f7911f90704 00007f7911f90692
0927 19:38:50.879339 (+112363us) log.cc:842] Serialized 506 byte log entry
0927 19:39:04.849126 (+13969787us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":691,"raft.queue_time_us":1022,"raft.run_cpu_time_us":1545,"raft.run_wall_time_us":1546,"spinlock_wait_cycles":1911147520,"thread_start_us":591,"threads_started":1,"wal-append.queue_time_us":45574,"wal-append.run_cpu_time_us":2133,"wal-append.run_wall_time_us":1115885}
I20260927 19:39:04.872498 2143 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40150 (request call id 1334) took 1847 ms. Trace:
I20260927 19:39:04.879739 2687 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:04.850520 5128 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:39:04.885527 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 277905us
I20260927 19:39:04.888289 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:39:04.891253 2110 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.749s (SENT) [suppressed 27 similar messages]
I20260927 19:39:04.845625 5123 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election won for term 7
W20260927 19:39:04.911854 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 52669.7us
I20260927 19:39:04.907109 5150 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.899531 5074 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.914914 4920 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:04.926570 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.938972 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.939407 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.939790 1992 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:04.929383 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.916878 2143 rpcz_store.cc:276] 0927 19:39:03.024693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:03.033935 (+ 9242us) service_pool.cc:227] Handling call
0927 19:39:04.773992 (+1740057us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:39:04.774596 (+ 604us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:04.872238 (+ 97642us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400263c50. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913961a09 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:39:04.872416 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4000854912}
I20260927 19:39:04.941293 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:04.943338 2654 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:04.944707 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:04.945171 2654 leader_election.cc:398] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:39:04.945864 2654 leader_election.cc:401] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:04.946764 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.940132 1994 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:39:04.947175 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.948195 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.948951 1994 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:04.904630 5055 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:04.915678 5144 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:04.954504 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.911105 5109 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:04.905282 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.956504 1961 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:04.958860 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.930038 5081 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.962345 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.962517 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.974161 1961 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:04.963388 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.945592 5074 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.969895 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.982470 2140 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:04.959182 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.924561 5128 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:04.976353 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.963625 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.977445 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:04.857659 5054 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:04.952646 5093 raft_consensus.cc:686] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:04.973480 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.949613 5121 raft_consensus.cc:686] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:04.984472 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.994402 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:04.961002 5123 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:04.986178 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.990517 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.958690 5169 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.968884 4963 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 7.
I20260927 19:39:04.995221 1990 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.007602 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.979588 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:04.963275 5144 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:04.991039 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:04.996399 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.971787 5109 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:04.982909 5000 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:04.942811 5150 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:04.970906 5158 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.002179 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:04.983341 4887 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:04.984788 5133 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:05.010860 5169 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:05.015654 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:05.000560 2138 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:05.036067 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.004372 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:05.017108 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:05.018406 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:05.037670 1994 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:05.038086 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:05.038082 1961 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:05.005913 2143 raft_consensus.cc:1261] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:05.021845 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.971390 5055 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.030679 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.046983 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.032318 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:04.981113 5092 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:05.032954 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.019766 5160 raft_consensus.cc:2726] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:05.052040 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:04.988467 4871 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260927 19:39:05.015261 5144 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:05.055200 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.022493 5090 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:05.001654 5086 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.042s
I20260927 19:39:05.070101 5147 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.022651 5096 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:05.071846 5096 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.072249 5147 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:05.069512 5172 log.cc:924] Time spent T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260927 19:39:05.073856 5096 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.073468 5121 consensus_queue.cc:237] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:05.074859 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 183630us
I20260927 19:39:05.067266 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:05.042762 5169 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.067962 2137 raft_consensus.cc:595] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:05.069249 5062 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:05.069621 5090 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:05.076926 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:05.029206 4887 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.079558 2137 raft_consensus.cc:603] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:05.083072 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:05.034860 5133 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.086153 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.052929 5056 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:05.087633 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:05.078714 5154 consensus_meta.cc:287] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.831s user 0.011s sys 0.001s
I20260927 19:39:05.083633 2142 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40164 (request call id 2615) took 1034 ms. Trace:
I20260927 19:39:05.088449 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:05.050534 5122 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:05.049831 5158 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:05.084065 5090 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.025074 5146 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260927 19:39:05.082731 5120 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:05.095695 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:05.080762 5113 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.086905 5157 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:05.087334 4887 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.098474 5120 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:05.107780 5120 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:05.097532 4863 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.114091 5153 consensus_queue.cc:1035] T 0fcc7a171b3e443d8f28fd70f8742470 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:05.115399 4863 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.115263 5120 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.098331 5093 consensus_queue.cc:237] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.117748 4887 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.101485 2110 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:05.120409 4996 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:05.120757 4996 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:05.121093 4996 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:05.121716 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.122902 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.123162 5150 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.124152 5142 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.089324 5088 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:05.128434 5088 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:05.128821 5088 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.131572 5137 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:05.094233 5174 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.143s user 0.001s sys 0.000s
I20260927 19:39:05.146092 2145 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.092038 5056 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:05.146770 5056 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:05.149730 5137 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:05.151235 4968 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.132159 4996 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.093647 2142 rpcz_store.cc:276] 0927 19:39:04.050081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:04.051141 (+ 1060us) service_pool.cc:227] Handling call
0927 19:39:04.742348 (+691207us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b5800004298. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02702 00007f79198f2b09 00007f791882e730 00007f79188f479b 00007f79188f4821 00007f79188b82b0 00007f79188a37f9 00007f79188a32a2 00007f7911f89684
0927 19:39:04.742558 (+ 210us) raft_consensus.cc:1457] Updating replica for 0 ops
0927 19:39:04.742758 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 8
0927 19:39:04.793984 (+ 51226us) log.cc:842] Serialized 14 byte log entry
0927 19:39:05.057336 (+263352us) log.cc:842] Serialized 14 byte log entry
0927 19:39:05.057738 (+ 402us) log.cc:842] Serialized 14 byte log entry
0927 19:39:05.067549 (+ 9811us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0927 19:39:05.067625 (+ 76us) raft_consensus.cc:1631] Marking committed up to 8
0927 19:39:05.067666 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:05.067734 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:05.083543 (+ 15809us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9406,"raft.queue_time_us":268,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":18472,"spinlock_wait_cycles":4177152,"thread_start_us":149553,"threads_started":2,"wal-append.queue_time_us":70990}
I20260927 19:39:05.119652 4887 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.136435 5120 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.157271 2136 raft_consensus.cc:1226] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:05.084597 5123 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.136543 2110 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:05.104782 5157 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:05.140369 5101 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.141232 5125 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:05.143639 5156 raft_consensus.cc:979] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:05.155823 5055 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.091279 5133 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:05.171694 4996 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.242125 1990 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 9: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:05.062107 1961 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:05.250224 1992 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:05.250769 1988 raft_consensus.cc:2379] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:05.258567 1959 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:05.264741 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 176993us
I20260927 19:39:05.294948 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.294871 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:39:05.299800 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 163139us
I20260927 19:39:05.301332 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.303726 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.299881 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.305141 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.166895 2685 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:05.318042 4871 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:05.196202 5081 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.203293 5105 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.154162 5056 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.358117 1996 raft_consensus.cc:1226] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260927 19:39:05.359057 5081 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.359277 5105 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.212267 5109 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.359833 5056 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.360280 2140 raft_consensus.cc:1226] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:05.360918 2143 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:05.171496 5124 consensus_queue.cc:1046] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:05.177976 5125 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:05.372898 5125 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:05.201418 5128 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.210845 5133 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.378108 2037 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:05.379997 5128 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.380098 5133 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.209509 5154 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33206 (request call id 40) took 3002 ms. Trace:
I20260927 19:39:05.181710 5156 raft_consensus.cc:1067] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:05.381163 5154 rpcz_store.cc:276] 0927 19:39:02.204598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.204735 (+ 137us) service_pool.cc:227] Handling call
0927 19:39:02.226916 (+ 22181us) log.cc:842] Serialized 501 byte log entry
0927 19:39:05.206594 (+2979678us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110919,"raft.run_cpu_time_us":1091,"raft.run_wall_time_us":1090,"thread_start_us":84800,"threads_started":1,"wal-append.queue_time_us":178164,"wal-append.run_cpu_time_us":9846,"wal-append.run_wall_time_us":1138044}
I20260927 19:39:05.384438 1935 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 228a1cf257434cbe9e5616dbd94e1864 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:39:05.172719 5157 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:05.095850 5158 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:05.214340 5169 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.332577 2145 raft_consensus.cc:2379] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.333336 5102 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:05.409937 2682 raft_consensus.cc:2379] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:05.336537 5101 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.338131 5021 consensus_queue.cc:1046] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:05.411306 5160 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:05.414397 5021 raft_consensus.cc:3032] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:39:05.341241 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.342676 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.343976 2136 raft_consensus.cc:2379] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 8: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:05.343518 5144 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.331977 5090 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.362207 5109 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:05.195366 5119 raft_consensus.cc:1013] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: attempt to promote peer 4e0335d1a85f421380029b95a91bf0c4: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260927 19:39:05.363862 5153 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260927 19:39:05.364387 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 106290us
I20260927 19:39:05.411283 5124 raft_consensus.cc:3032] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:05.442200 5124 raft_consensus.cc:729] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:05.440466 2683 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEARNER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260927 19:39:05.366659 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 304473us
I20260927 19:39:05.443682 5124 consensus_queue.cc:260] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.366757 4887 raft_consensus.cc:2726] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:05.445176 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:05.445770 1893 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:05.387372 5055 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:05.448659 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.449110 5055 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.451308 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.451540 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.451835 5055 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.381179 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:05.452723 5133 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260927 19:39:05.453521 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 72544us
I20260927 19:39:05.454144 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.455674 4871 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:05.409173 2687 raft_consensus.cc:2354] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:05.409277 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.448340 1893 catalog_manager.cc:5495] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 5 to 6, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:05.411830 5158 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:05.418186 2141 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.419333 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:05.423296 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 51194.7us
I20260927 19:39:05.415009 5021 raft_consensus.cc:729] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
W20260927 19:39:05.429131 1935 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 44939.6us
I20260927 19:39:05.432838 2144 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.435307 5090 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:05.436064 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 59458.2us
I20260927 19:39:05.439257 2139 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260927 19:39:05.453198 5133 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.473637 5133 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:05.477867 2136 raft_consensus.cc:2354] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8.
I20260927 19:39:05.487481 5169 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.488611 4968 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.489398 5144 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:05.490077 5021 consensus_queue.cc:260] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.487275 5118 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260927 19:39:05.498553 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.504717 2067 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:39:05.498792 5124 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:05.494036 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.510840 2070 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:05.493294 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.512809 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.514639 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83f97251c21d4e979bce8419c208369c"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:05.515619 2136 raft_consensus.cc:595] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:05.516067 2136 raft_consensus.cc:603] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:05.518239 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.518539 5133 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.520715 5133 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:05.521538 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 48993.2us
I20260927 19:39:05.521992 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260927 19:39:05.527192 1988 raft_consensus.cc:1226] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:05.526926 5169 consensus_queue.cc:1046] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260927 19:39:05.528472 5169 raft_consensus.cc:3032] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 19:39:05.528841 1989 raft_consensus.cc:1226] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:05.528893 5169 raft_consensus.cc:729] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:05.535511 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.536401 2067 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:05.536230 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.537679 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.537914 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.538007 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.538722 2064 raft_consensus.cc:3037] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:05.494900 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.540776 2068 raft_consensus.cc:2379] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 7: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:39:05.542896 5074 consensus_meta.cc:287] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.547s user 0.008s sys 0.008s
I20260927 19:39:05.538447 2067 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.543495 5074 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.534878 5169 consensus_queue.cc:260] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:05.559742 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 53214.6us
I20260927 19:39:05.564445 2111 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:05.565734 1942 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 69f09866d0a1477585f05b2f82ff4c67 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:05.546602 5169 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:05.582376 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.583794 2064 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:05.594702 5123 raft_consensus.cc:2726] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:05.597724 2679 raft_consensus.cc:2379] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.599610 2034 leader_election.cc:398] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:05.600016 2034 leader_election.cc:401] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:05.601218 5109 consensus_queue.cc:1035] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260927 19:39:05.591010 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:05.608999 2067 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:39:05.609751 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.606143 1962 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:05.580353 2069 raft_consensus.cc:2932] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Committing config change with OpId 8.8: config changed from index 4 to 8, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:05.617635 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.546700 5125 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.619092 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.623098 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.630059 2687 raft_consensus.cc:2379] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:05.614192 5055 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:05.613917 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260927 19:39:05.637435 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 52185.9us
I20260927 19:39:05.624073 5074 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.637137 1893 catalog_manager.cc:5495] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:05.651870 2071 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:05.644151 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:05.651023 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:05.659281 2067 raft_consensus.cc:1261] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260927 19:39:05.660581 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.661528 2688 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:05.664522 1996 raft_consensus.cc:2354] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:05.647769 5090 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260927 19:39:05.634101 5123 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.661706 2067 pending_rounds.cc:76] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 5
I20260927 19:39:05.668557 1990 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.668406 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:05.669117 5105 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260927 19:39:05.669555 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 105290us
I20260927 19:39:05.669981 1962 leader_election.cc:398] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8.
I20260927 19:39:05.673224 1992 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:05.674755 1962 leader_election.cc:401] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260927 19:39:05.674882 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:05.644232 5124 consensus_queue.cc:1046] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260927 19:39:05.657743 5118 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.674242 2067 pending_rounds.cc:97] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260927 19:39:05.677495 5124 raft_consensus.cc:3032] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:05.665231 5021 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:05.686676 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.679802 2067 raft_consensus.cc:2866] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:05.693259 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.660532 5088 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.700722 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.681218 5124 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:05.703226 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:05.677347 5118 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.708182 2138 raft_consensus.cc:2354] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:39:05.709059 2145 raft_consensus.cc:2379] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:05.707083 5123 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.713127 2111 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:05.695897 5074 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:05.694770 5157 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.726130 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.718305 5063 consensus_queue.cc:1046] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260927 19:39:05.736824 2070 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 9: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:05.732581 5063 raft_consensus.cc:3032] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:05.732733 5088 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.746749 5118 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.749951 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.762292 2067 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.752815 5074 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.753870 5124 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.754410 5125 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.765666 1962 leader_election.cc:398] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:39:05.767131 1962 leader_election.cc:401] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260927 19:39:05.771035 2065 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:39:05.771474 1893 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 2 to 7, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:05.774360 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.749835 5063 raft_consensus.cc:729] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:05.777781 1962 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:05.778349 1990 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:05.765779 5081 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:05.783324 5056 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8.
I20260927 19:39:05.802994 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.803874 2063 raft_consensus.cc:2379] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:05.784757 5081 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:05.803285 2145 raft_consensus.cc:1261] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:39:05.800992 5124 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:05.797299 5123 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:05.824671 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:05.827308 2144 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:05.808971 5081 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.829131 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.829689 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.833570 2111 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:05.835270 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.832530 1992 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:05.839344 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.842474 2071 raft_consensus.cc:2379] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:05.837872 5118 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:39:05.848569 2679 raft_consensus.cc:2454] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:39:05.819341 5105 consensus_queue.cc:1035] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260927 19:39:05.855243 1962 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:05.859159 2070 raft_consensus.cc:2379] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:05.859864 4887 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:05.860385 4887 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.860766 4887 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:05.864493 2067 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:05.854269 5123 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:05.867305 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.873198 5093 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:05.873521 5074 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.858394 5157 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:05.878247 2108 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:05.871444 5123 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:05.889500 2140 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:05.858259 5158 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.891198 2071 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:05.897148 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.898000 2070 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:05.887785 5123 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
W20260927 19:39:05.897467 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d0f797e07d634f6abe29556a11863581, TS 4cc2df048f424e08803a390946004df0 -> TS 4e0335d1a85f421380029b95a91bf0c4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.903811 2111 leader_election.cc:398] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:05.904474 2111 leader_election.cc:401] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260927 19:39:05.887408 1893 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 5 to 7, term changed from 5 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:39:05.904379 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d0f797e07d634f6abe29556a11863581, TS 4cc2df048f424e08803a390946004df0 -> TS 4e0335d1a85f421380029b95a91bf0c4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:05.861637 5063 consensus_queue.cc:260] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.910454 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:05.913452 5101 consensus_queue.cc:1046] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260927 19:39:05.913930 5074 raft_consensus.cc:2781] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:39:05.924033 5137 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:05.926882 1893 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:05.932746 5109 raft_consensus.cc:3032] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:05.933312 5109 raft_consensus.cc:729] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:05.941112 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.942041 2064 raft_consensus.cc:2379] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:05.942092 5074 raft_consensus.cc:686] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:05.942958 2068 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 19:39:05.913108 5056 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:05.942121 2063 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:05.943223 5074 consensus_queue.cc:237] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:05.944350 5056 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:05.945776 5187 log.cc:924] Time spent T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20260927 19:39:05.946779 5056 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:05.941004 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:05.949736 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:05.950971 5063 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:05.901666 5153 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260927 19:39:05.959470 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.962247 1893 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:05.912617 5093 raft_consensus.cc:979] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:39:05.970002 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:05.962302 2111 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:05.914527 5133 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:06.000288 2067 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:06.001909 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:06.008751 5193 log.cc:924] Time spent T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
I20260927 19:39:06.010586 5093 raft_consensus.cc:1067] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:39:06.012166 5137 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for earlier term 8. Current term is 9.
I20260927 19:39:06.013819 5137 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:06.014312 5137 raft_consensus.cc:686] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:06.020637 4887 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.022943 4887 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.024312 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.024976 2144 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:39:06.027004 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 112908us
W20260927 19:39:06.040691 1942 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ac5251fc915042bab4ce94b1c17deb87, TS 69f09866d0a1477585f05b2f82ff4c67 -> TS 4e0335d1a85f421380029b95a91bf0c4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:06.041144 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ac5251fc915042bab4ce94b1c17deb87, TS 69f09866d0a1477585f05b2f82ff4c67 -> TS 4e0335d1a85f421380029b95a91bf0c4 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.027137 5109 consensus_queue.cc:260] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.046047 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 242) took 9137 ms. Trace:
I20260927 19:39:06.046907 5109 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:06.045828 5118 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Committing config change with OpId 8.9: config changed from index 6 to 9, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:06.051664 2680 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 6 to 9, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:06.015276 5137 consensus_queue.cc:237] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.058276 4887 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:06.059145 4887 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.062058 4887 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.062592 5102 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:06.063565 5102 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.008802 5133 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.068073 5133 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.062695 5056 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:06.073556 2145 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:39:06.074345 2145 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:06.074796 2145 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:06.082911 2069 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:06.088045 2144 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:06.083743 2069 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:06.095209 2069 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:06.098165 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:06.102047 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.098179 1962 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:06.092017 5123 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.107625 5102 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.108247 5158 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.109579 5123 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:06.109448 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:06.115262 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 153049us
I20260927 19:39:06.116393 2111 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:06.123353 4887 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:06.124159 4887 raft_consensus.cc:686] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:06.125159 4887 consensus_queue.cc:237] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.046500 1892 rpcz_store.cc:276] 0927 19:38:56.908888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:38:56.909451 (+ 563us) service_pool.cc:227] Handling call
0927 19:38:56.989439 (+ 79988us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b54002f58d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79198bcd6c 00007f79199b6c3e 00007f7917cc8fb4 00007f7917cc8f42 00007f7917cc8e71 00007f7917cc8dec 00007f7917cc80a2 00007f7911d8c94c 00007f7911d8bd86
0927 19:39:06.045818 (+9056379us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:02.980356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:03.049198 (+ 68842us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:03.049236 (+ 38us) write_op.cc:435] Acquired schema lock
0927 19:39:03.049255 (+ 19us) tablet.cc:660] Decoding operations
0927 19:39:03.049928 (+ 673us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:03.049985 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:03.050013 (+ 28us) tablet.cc:683] Acquiring locks for 5 operations
0927 19:39:03.050553 (+ 540us) tablet.cc:699] Row locks acquired
0927 19:39:03.050574 (+ 21us) write_op.cc:260] PREPARE: finished
0927 19:39:03.050726 (+ 152us) write_op.cc:270] Start()
0927 19:39:03.050836 (+ 110us) write_op.cc:276] Timestamp: P: 1790537943050668 usec, L: 0
0927 19:39:03.050859 (+ 23us) op_driver.cc:331] REPLICATION: starting
0927 19:39:03.051466 (+ 607us) log.cc:842] Serialized 2089 byte log entry
0927 19:39:03.714554 (+663088us) op_driver.cc:444] REPLICATION: finished
0927 19:39:04.050777 (+336223us) write_op.cc:301] APPLY: starting
0927 19:39:04.057466 (+ 6689us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:04.093673 (+ 36207us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:04.106721 (+ 13048us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:04.172931 (+ 66210us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:04.205127 (+ 32196us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0927 19:39:04.212123 (+ 6996us) write_op.cc:312] APPLY: finished
0927 19:39:04.243879 (+ 31756us) log.cc:842] Serialized 47 byte log entry
0927 19:39:04.244082 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:04.244185 (+ 103us) write_op.cc:454] Released schema lock
0927 19:39:04.244240 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:04.244509 (+ 269us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:04.244536 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8858,"reported_tablets":36,"spinlock_wait_cycles":15504128,"child_traces":[["op",{"apply.queue_time_us":305694,"apply.run_cpu_time_us":3859,"apply.run_wall_time_us":210357,"num_ops":5,"prepare.queue_time_us":335782,"prepare.run_cpu_time_us":3533,"prepare.run_wall_time_us":82812,"replication_time_us":631757,"thread_start_us":433148,"threads_started":2}]]}
I20260927 19:39:06.135083 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:06.074361 5056 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.140487 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.142208 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:06.143479 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.143620 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.144397 2065 raft_consensus.cc:2354] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 8. Current term is 9.
I20260927 19:39:06.144734 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.145504 2068 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:06.144812 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.144407 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.150233 2062 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:06.152804 1992 raft_consensus.cc:1261] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:06.149448 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.155244 2064 raft_consensus.cc:2354] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8.
I20260927 19:39:06.065335 5081 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.131675 5074 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:06.165617 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.167213 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.167889 2071 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:06.140213 2138 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 6 to 9, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:06.170683 5081 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:06.176543 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.170324 2069 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.172605 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.178417 2062 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:06.178511 1993 raft_consensus.cc:3037] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:06.174029 5093 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:06.178860 2062 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:06.175961 2066 raft_consensus.cc:2379] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:06.180249 5093 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.182096 5158 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:06.182703 5093 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.175715 5074 raft_consensus.cc:686] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:06.184350 5074 consensus_queue.cc:237] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.186707 5056 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.188160 2145 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.189416 2065 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260927 19:39:06.182775 5158 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.193863 1987 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 19:39:06.194247 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:06.198246 2136 raft_consensus.cc:1226] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 3. Current term is 4. Ops: []
I20260927 19:39:06.200307 2069 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.193899 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:06.213196 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.215462 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.215713 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:06.216225 2064 raft_consensus.cc:2354] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:39:06.212466 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.220881 2145 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.227718 1962 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.221153 2062 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.229996 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:06.232311 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:06.232908 2682 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:06.235980 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:06.236922 2064 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:06.237938 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:06.240044 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:06.243580 5105 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.248541 2070 raft_consensus.cc:1226] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 9. Ops: []
I20260927 19:39:06.251577 5158 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.258540 5153 consensus_queue.cc:1046] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:06.260038 5153 raft_consensus.cc:3032] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Stepping down as leader of term 3
I20260927 19:39:06.260449 5153 raft_consensus.cc:729] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:06.261723 5153 consensus_queue.cc:260] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.263463 5153 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 3 FOLLOWER]: Advancing to term 4
I20260927 19:39:06.268657 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.270124 2063 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:06.232575 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.295006 2069 raft_consensus.cc:1261] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260927 19:39:06.296785 2140 raft_consensus.cc:2379] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 6: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:06.281288 5158 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:06.315737 2071 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:06.320721 2070 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:06.334143 2145 raft_consensus.cc:1226] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Rejecting Update request from peer 4e0335d1a85f421380029b95a91bf0c4 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:06.336436 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.336586 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.337209 1995 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:06.337239 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.338168 2144 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:06.349278 2068 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:06.336833 2111 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:06.333214 1993 raft_consensus.cc:2454] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
W20260927 19:39:06.350886 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 121100us
I20260927 19:39:06.350886 5101 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260927 19:39:06.350570 1989 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:06.352192 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.355924 5109 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:39:06.359366 5101 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:39:06.365697 2111 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:06.376748 2062 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:06.378382 2682 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:06.380738 2108 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
W20260927 19:39:06.365510 5196 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260927 19:39:06.379145 5157 raft_consensus.cc:2726] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:06.384945 5123 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:39:06.390116 5123 raft_consensus.cc:686] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:06.391145 5123 consensus_queue.cc:237] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.393424 2037 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.382158 5101 raft_consensus.cc:2781] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:06.403950 5158 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:06.405045 5101 raft_consensus.cc:686] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:06.428354 1995 raft_consensus.cc:2454] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:06.429492 5101 consensus_queue.cc:237] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:06.442379 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 57627us
I20260927 19:39:06.450337 2037 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.451367 5158 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:06.452886 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.453841 5158 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.454063 2682 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:39:06.461279 5074 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:06.463119 5123 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:06.463761 5123 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.466897 5123 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.468124 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:39:06.470532 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 89708.7us
I20260927 19:39:06.474241 5063 consensus_queue.cc:1046] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:39:06.478021 2111 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:06.486187 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.494292 2111 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:06.483444 5123 raft_consensus.cc:2781] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:06.502154 5109 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:06.502518 5158 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:39:06.523192 5123 raft_consensus.cc:686] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:06.531016 5123 consensus_queue.cc:237] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:06.533263 5063 raft_consensus.cc:3032] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:06.533844 5063 raft_consensus.cc:729] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:06.535048 5063 consensus_queue.cc:260] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.529448 2037 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.537462 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.537904 5157 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:06.539170 5120 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:06.537107 5063 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:06.544454 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.563071 5158 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:06.565713 2037 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:06.570811 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 69941.2us
I20260927 19:39:06.571085 5158 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.580641 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.581779 5157 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.584379 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.585304 5157 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:06.587216 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.596267 5158 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:06.608368 2037 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.611928 5158 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:06.613626 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.616161 5158 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:06.617722 1989 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:06.619479 5137 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:06.621438 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.632656 5137 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:06.633555 5137 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.637856 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:06.638535 2679 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 10.
I20260927 19:39:06.641714 5158 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:06.644032 2108 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:06.651130 5101 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:06.651566 5101 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:06.651923 5101 raft_consensus.cc:3037] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:06.653525 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.660391 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.660846 5101 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.667742 1987 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 10: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
W20260927 19:39:06.668867 5227 log.cc:924] Time spent T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260927 19:39:06.670897 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.671502 1987 raft_consensus.cc:2379] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 10: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:06.673237 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:06.635879 5137 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.669575 5101 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.693517 2679 raft_consensus.cc:2454] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 10.
I20260927 19:39:06.694833 2108 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
W20260927 19:39:06.699753 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 46301.6us
I20260927 19:39:06.684549 5158 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.700986 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 158) took 4299 ms. Trace:
I20260927 19:39:06.699656 5102 raft_consensus.cc:2781] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 19:39:06.704689 5102 raft_consensus.cc:686] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:06.705900 5102 consensus_queue.cc:237] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.705922 1891 catalog_manager.cc:5495] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:06.702786 1893 rpcz_store.cc:276] 0927 19:39:02.399807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:02.400082 (+ 275us) service_pool.cc:227] Handling call
0927 19:39:02.409523 (+ 9441us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b5400000910. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f7910a02851 00007f7910a02d92 00007f79199537c9 00007f79198ebc36 00007f79198bcc28 00007f79199b6c3e 00007f7917cc8fb4 00007f7917cc8f42 00007f7917cc8e71 00007f7917cc8dec
0927 19:39:06.699329 (+4289806us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:06.123542 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:06.124523 (+ 981us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:06.124583 (+ 60us) write_op.cc:435] Acquired schema lock
0927 19:39:06.125822 (+ 1239us) tablet.cc:660] Decoding operations
0927 19:39:06.126995 (+ 1173us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:06.127134 (+ 139us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:06.127188 (+ 54us) tablet.cc:683] Acquiring locks for 7 operations
0927 19:39:06.128033 (+ 845us) tablet.cc:699] Row locks acquired
0927 19:39:06.128074 (+ 41us) write_op.cc:260] PREPARE: finished
0927 19:39:06.128278 (+ 204us) write_op.cc:270] Start()
0927 19:39:06.140438 (+ 12160us) write_op.cc:276] Timestamp: P: 1790537946128199 usec, L: 0
0927 19:39:06.163805 (+ 23367us) op_driver.cc:331] REPLICATION: starting
0927 19:39:06.350178 (+186373us) log.cc:842] Serialized 2894 byte log entry
0927 19:39:06.556034 (+205856us) op_driver.cc:444] REPLICATION: finished
0927 19:39:06.629137 (+ 73103us) write_op.cc:301] APPLY: starting
0927 19:39:06.629239 (+ 102us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:06.629483 (+ 244us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:06.629611 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:06.630587 (+ 976us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:06.630857 (+ 270us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0927 19:39:06.630918 (+ 61us) write_op.cc:312] APPLY: finished
0927 19:39:06.631264 (+ 346us) log.cc:842] Serialized 59 byte log entry
0927 19:39:06.631440 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:06.631558 (+ 118us) write_op.cc:454] Released schema lock
0927 19:39:06.631616 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:06.631980 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:06.632012 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1076530,"reported_tablets":10,"spinlock_wait_cycles":4913024,"child_traces":[["op",{"apply.queue_time_us":70682,"apply.run_cpu_time_us":3286,"apply.run_wall_time_us":3331,"num_ops":7,"prepare.queue_time_us":16450,"prepare.run_cpu_time_us":15343,"prepare.run_wall_time_us":294770,"raft.queue_time_us":107414,"raft.run_cpu_time_us":16505,"raft.run_wall_time_us":40162,"replication_time_us":392135,"thread_start_us":165485,"threads_started":4,"wal-append.queue_time_us":84493}]]}
I20260927 19:39:06.712658 1891 catalog_manager.cc:5495] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:06.723256 1987 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:39:06.723342 1891 catalog_manager.cc:5495] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:06.732340 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:06.736590 5074 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:06.724103 5158 consensus_queue.cc:1046] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260927 19:39:06.745743 2136 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260927 19:39:06.750022 5157 raft_consensus.cc:2741] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260927 19:39:06.752802 4864 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:06.789563 1990 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260927 19:39:06.781603 1891 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 6, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129), now has a pending config: VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129), VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130), VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131), NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:39:06.798818 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.798959 5157 raft_consensus.cc:3032] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:06.800830 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.850018 5157 raft_consensus.cc:729] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:06.841738 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.847216 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.851238 5157 consensus_queue.cc:260] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.852504 2136 raft_consensus.cc:2354] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:06.853223 5157 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:06.856132 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.864140 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.865084 1987 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 9: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:06.867053 1891 catalog_manager.cc:5495] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 6, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:06.871570 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.875111 1987 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:06.882936 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.887406 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.892449 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:06.894737 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:06.902143 2679 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:39:06.904481 5102 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:06.912022 1993 raft_consensus.cc:1226] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 3. Current term is 4. Ops: []
I20260927 19:39:06.915174 2653 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:06.916620 5063 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260927 19:39:06.922590 1993 raft_consensus.cc:1261] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:39:06.924327 5206 consensus_queue.cc:1035] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:06.954133 2653 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.955049 5222 raft_consensus.cc:2781] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:06.955323 2653 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.955636 5222 raft_consensus.cc:686] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:06.956403 5223 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:06.956897 5223 raft_consensus.cc:686] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:06.957782 5223 consensus_queue.cc:237] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.958580 2110 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:06.958858 5137 consensus_queue.cc:1035] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:39:06.959219 5081 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:06.960049 2110 leader_election.cc:302] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:06.956630 5222 consensus_queue.cc:237] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.970984 1961 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:06.972697 4979 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.972851 1961 leader_election.cc:398] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 8. Current term is 9.
I20260927 19:39:06.972994 5123 raft_consensus.cc:2726] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:39:06.973728 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.975823 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.973179 1961 leader_election.cc:401] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260927 19:39:06.977442 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.978458 4979 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 8. Current term is 9.
I20260927 19:39:06.978669 1961 leader_election.cc:398] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8.
I20260927 19:39:06.979116 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.979489 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:06.980262 4979 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:39:06.980654 1961 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:06.987687 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.986577 5120 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:06.993939 5120 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:06.995992 5120 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:06.996903 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:06.990408 5137 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260927 19:39:06.997542 2682 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:06.995899 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.998579 1987 raft_consensus.cc:3032] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:06.997825 1890 catalog_manager.cc:5495] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 5, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:06.990553 5101 consensus_queue.cc:1035] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260927 19:39:06.990909 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:06.999506 1890 catalog_manager.cc:5495] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 5 to 6, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 6 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:06.986123 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:06.989846 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:06.993137 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:07.001200 1890 catalog_manager.cc:5495] T f61659c573e142aa9c3fbb673f3e15f0 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 5, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:07.002092 5101 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:07.002734 1890 catalog_manager.cc:5495] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 1 to 6, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:07.003261 2110 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:07.003556 2034 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:07.004256 5101 raft_consensus.cc:2726] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:39:07.004493 5120 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:07.004106 1890 catalog_manager.cc:5495] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 3 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:07.004842 5120 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:07.005453 5120 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:07.006645 1961 leader_election.cc:398] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6.
I20260927 19:39:07.006559 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60280
I20260927 19:39:06.993494 4979 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:07.007161 1990 raft_consensus.cc:595] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Received request to transfer leadership
I20260927 19:39:07.007702 4864 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 19:39:07.008064 1961 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:07.009019 5118 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:06.998906 1987 raft_consensus.cc:729] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:07.005410 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:07.011061 1987 consensus_queue.cc:260] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.014387 1987 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 9
I20260927 19:39:07.023667 5118 raft_consensus.cc:979] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:39:07.026970 1987 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:07.009147 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.024215 5118 raft_consensus.cc:1067] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Signalling peer 4e0335d1a85f421380029b95a91bf0c4 to start an election
I20260927 19:39:07.032011 4864 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:07.032272 2682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
from {username='slave'} at 127.0.0.1:34936
I20260927 19:39:07.032385 4864 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:07.032891 2682 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:07.033305 2682 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:07.033520 5101 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:07.035378 1890 catalog_manager.cc:5495] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
W20260927 19:39:07.036705 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 50402.9us
I20260927 19:39:07.037511 1890 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 reported cstate change: config changed from index -1 to 6, term changed from 6 to 8, NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) added, no longer has a pending config: VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129), VOTER 4cc2df048f424e08803a390946004df0 (127.0.204.130), VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131), NON_VOTER 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } } }
I20260927 19:39:07.032892 4864 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:07.040042 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:07.040671 2136 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:07.040159 1890 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 5 to 7, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:07.041728 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:07.042703 5157 consensus_queue.cc:1046] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:39:07.042972 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:07.043762 5157 raft_consensus.cc:3032] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:07.044119 5157 raft_consensus.cc:729] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:07.044582 2137 raft_consensus.cc:1226] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260927 19:39:07.045249 5157 consensus_queue.cc:260] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.046074 5233 consensus_queue.cc:1046] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:39:07.046516 5157 raft_consensus.cc:3037] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:07.046880 5233 consensus_queue.cc:1225] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer 3d1bc2cd9efe43399ddba396f5ddff85 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260927 19:39:07.046957 1961 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.047835 4887 raft_consensus.cc:2726] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:07.048066 5158 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:07.048588 5158 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.048285 1961 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:07.054133 5118 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader election won for term 4
I20260927 19:39:07.054723 1961 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:07.054765 5118 raft_consensus.cc:686] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:07.057430 1961 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.043053 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60280
I20260927 19:39:07.062552 1990 raft_consensus.cc:595] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Received request to transfer leadership
I20260927 19:39:07.062963 5133 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:07.063748 5133 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:07.064893 5133 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.068444 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60280
I20260927 19:39:07.069464 1990 raft_consensus.cc:595] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Received request to transfer leadership
I20260927 19:39:07.060228 2682 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.059237 1890 catalog_manager.cc:5495] T 39d534ed586444279970bb03de60c919 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:07.067713 4864 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:07.072335 1942 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 51dd0aa486b149cb9fc4a9dd54e3cdf4 already in progress
W20260927 19:39:07.072793 1942 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 51dd0aa486b149cb9fc4a9dd54e3cdf4 already in progress
I20260927 19:39:07.068800 4887 consensus_queue.cc:1035] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260927 19:39:07.077715 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:07.078480 1990 raft_consensus.cc:3032] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:07.078853 1990 raft_consensus.cc:729] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:07.071679 4979 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:07.081277 5158 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:07.081948 5158 raft_consensus.cc:3032] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:07.087044 1987 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260927 19:39:07.082346 5158 raft_consensus.cc:729] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:07.094765 2136 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:07.096115 1995 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:07.096993 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:07.099721 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 258) took 6476 ms. Trace:
I20260927 19:39:07.100217 1891 rpcz_store.cc:276] 0927 19:39:00.623022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:00.631865 (+ 8843us) service_pool.cc:227] Handling call
0927 19:39:06.938881 (+6307016us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:06.938912 (+ 31us) write_op.cc:454] Released schema lock
0927 19:39:07.099645 (+160733us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:06.918066 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:06.918628 (+ 562us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:06.918661 (+ 33us) write_op.cc:435] Acquired schema lock
0927 19:39:06.918679 (+ 18us) tablet.cc:660] Decoding operations
0927 19:39:06.919384 (+ 705us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:06.919444 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:06.919479 (+ 35us) tablet.cc:683] Acquiring locks for 5 operations
0927 19:39:06.920064 (+ 585us) tablet.cc:699] Row locks acquired
0927 19:39:06.920081 (+ 17us) write_op.cc:260] PREPARE: finished
0927 19:39:06.920246 (+ 165us) write_op.cc:270] Start()
0927 19:39:06.920350 (+ 104us) write_op.cc:276] Timestamp: P: 1790537946920200 usec, L: 0
0927 19:39:06.920372 (+ 22us) op_driver.cc:331] REPLICATION: starting
0927 19:39:06.920991 (+ 619us) log.cc:842] Serialized 2345 byte log entry
0927 19:39:06.923196 (+ 2205us) op_driver.cc:444] REPLICATION: finished
0927 19:39:06.923618 (+ 422us) write_op.cc:301] APPLY: starting
0927 19:39:06.923697 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:06.923855 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:06.923960 (+ 105us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:06.924602 (+ 642us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:06.924781 (+ 179us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0927 19:39:06.924838 (+ 57us) write_op.cc:312] APPLY: finished
0927 19:39:06.925057 (+ 219us) log.cc:842] Serialized 47 byte log entry
0927 19:39:06.925227 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:06.925333 (+ 106us) write_op.cc:454] Released schema lock
0927 19:39:06.925403 (+ 70us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":156962,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":222,"apply.run_cpu_time_us":2057,"apply.run_wall_time_us":2056,"num_ops":5,"prepare.queue_time_us":19968,"prepare.run_cpu_time_us":3223,"prepare.run_wall_time_us":7931,"replication_time_us":2713}]]}
I20260927 19:39:07.041416 5120 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.082363 5056 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260927 19:39:07.106348 5120 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:07.106997 5120 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:07.107542 5120 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.111095 5118 consensus_queue.cc:237] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.112738 5123 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:39:07.113710 5109 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:39:07.118273 2680 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260927 19:39:07.109959 5158 consensus_queue.cc:260] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.119724 5102 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 19:39:07.120044 5158 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:07.123477 2682 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:07.079865 1990 consensus_queue.cc:260] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.125766 1990 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
W20260927 19:39:07.127638 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 83077.4us
I20260927 19:39:07.131630 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:07.131654 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:07.133065 2137 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:07.141812 5109 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:07.142640 5109 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.151794 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:07.151840 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:07.153797 1987 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
I20260927 19:39:07.156728 1990 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:07.158295 2111 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:07.138954 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:07.165733 2679 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:07.169059 2034 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:07.156371 5109 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:07.167297 5101 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:07.191461 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.191848 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.192500 2062 raft_consensus.cc:2362] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6. Re-sending same reply.
I20260927 19:39:07.196022 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.196952 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.197188 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.198053 2065 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:07.198424 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.181917 5120 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:07.200773 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.202395 2070 raft_consensus.cc:2354] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 8.
I20260927 19:39:07.212392 2064 raft_consensus.cc:2354] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 7. Current term is 8.
I20260927 19:39:07.198853 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:07.214722 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.210623 2137 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:07.221737 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.222831 2654 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:07.199144 2066 raft_consensus.cc:2362] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8. Re-sending same reply.
I20260927 19:39:07.226557 1993 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:07.193933 5101 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:07.231276 2069 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:07.195233 5233 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:39:07.234329 2062 raft_consensus.cc:2379] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 4: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:07.238309 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.238794 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.196278 5243 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:07.239593 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.240801 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.241930 2063 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 9: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:07.254175 2065 raft_consensus.cc:1226] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:07.301949 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.273664 2062 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260927 19:39:07.300331 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.303028 2070 raft_consensus.cc:2362] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8. Re-sending same reply.
I20260927 19:39:07.303355 2066 raft_consensus.cc:1799] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Handling vote request from an unknown peer 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.274263 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.237533 5101 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:07.314832 2066 raft_consensus.cc:2396] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260927 19:39:07.315783 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.317441 2065 raft_consensus.cc:3037] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:07.318264 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.307845 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.319264 2063 raft_consensus.cc:2379] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 9: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:07.319775 2062 raft_consensus.cc:2379] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:07.233789 5120 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:07.316542 2068 raft_consensus.cc:2362] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8. Re-sending same reply.
I20260927 19:39:07.309183 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.242774 5063 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:07.301875 5243 raft_consensus.cc:686] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:07.321952 2070 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:07.325963 2071 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:07.322063 2067 raft_consensus.cc:2396] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260927 19:39:07.326581 2110 leader_election.cc:302] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:07.334203 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.270920 5233 raft_consensus.cc:686] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:07.342623 2653 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:07.325557 4864 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:07.336496 2069 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:07.339941 2037 leader_election.cc:302] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.341338 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:07.343814 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:07.320456 2064 raft_consensus.cc:2396] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260927 19:39:07.344388 2137 raft_consensus.cc:1226] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:07.346453 2062 raft_consensus.cc:1261] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260927 19:39:07.301735 5206 consensus_queue.cc:1046] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260927 19:39:07.354076 2063 raft_consensus.cc:2379] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:07.355171 2068 raft_consensus.cc:2379] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 4: Already voted for candidate 4cc2df048f424e08803a390946004df0 in this term.
I20260927 19:39:07.355891 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:07.398159 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 58288.3us
I20260927 19:39:07.401342 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:07.401465 2037 leader_election.cc:398] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:07.358537 5123 raft_consensus.cc:2726] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:07.372365 5206 raft_consensus.cc:2726] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260927 19:39:07.419358 5242 log.cc:924] Time spent T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20260927 19:39:07.401419 5223 raft_consensus.cc:2726] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:07.402547 5074 consensus_queue.cc:1046] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260927 19:39:07.424392 5137 raft_consensus.cc:3032] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:07.424860 5137 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:07.426731 2071 raft_consensus.cc:1261] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260927 19:39:07.427088 2071 pending_rounds.cc:76] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 4
I20260927 19:39:07.427318 2071 pending_rounds.cc:97] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260927 19:39:07.427543 2071 raft_consensus.cc:2866] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260927 19:39:07.434024 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 91473.3us
I20260927 19:39:07.353924 5120 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.428696 2065 raft_consensus.cc:1261] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:39:07.444510 2653 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.440582 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.429265 5123 raft_consensus.cc:3032] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Stepping down as leader of term 5
W20260927 19:39:07.457938 5251 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260927 19:39:07.446592 5257 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:07.500913 2068 raft_consensus.cc:1226] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260927 19:39:07.471073 5222 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:07.470383 5123 raft_consensus.cc:729] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:07.531859 5262 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:07.532526 5262 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:07.533725 5250 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260927 19:39:07.535015 5262 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:07.535889 5262 raft_consensus.cc:2726] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:07.443326 5137 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.538341 5137 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
W20260927 19:39:07.541679 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 106879us
I20260927 19:39:07.532912 5248 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:07.542685 2653 leader_election.cc:302] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:07.542799 5248 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:07.543426 5248 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:07.544450 5124 raft_consensus.cc:2726] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:07.546667 2653 leader_election.cc:398] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 7. Current term is 8.
I20260927 19:39:07.546985 2653 leader_election.cc:401] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:07.549502 2653 leader_election.cc:398] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 6. Current term is 8.
I20260927 19:39:07.551985 2070 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:07.555152 2069 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260927 19:39:07.563019 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 198614us
I20260927 19:39:07.577301 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:07.554754 5124 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515) with higher term. Message: Invalid argument: T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 7. Current term is 8.
I20260927 19:39:07.446262 5233 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.574132 5063 raft_consensus.cc:686] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:07.455569 5243 consensus_queue.cc:237] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.601749 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:07.613901 1987 raft_consensus.cc:2379] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:07.604925 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:07.552430 5222 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.625883 1990 raft_consensus.cc:2379] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 5: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:07.615921 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:07.618101 5124 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:07.629834 5124 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.617825 5063 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.632560 5124 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:07.636601 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:07.549451 5257 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:07.641738 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 92356.5us
I20260927 19:39:07.638927 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:07.590862 5206 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260927 19:39:07.660220 1990 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:39:07.688264 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:07.699805 1989 raft_consensus.cc:1261] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260927 19:39:07.717901 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 127323us
I20260927 19:39:07.730208 5124 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:07.621222 5123 consensus_queue.cc:260] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:07.756982 5123 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:07.749140 5157 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:07.758121 5157 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:07.760236 5157 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:07.745131 5120 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:07.778802 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 119076us
I20260927 19:39:07.754122 5124 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:07.786820 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:39:07.785778 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 170223us
I20260927 19:39:07.820057 5101 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:07.821816 5158 consensus_meta.cc:287] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0: Time spent flushing consensus metadata: real 0.701s user 0.013s sys 0.000s
I20260927 19:39:07.826673 2067 raft_consensus.cc:1261] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260927 19:39:07.796435 5124 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:07.833734 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:07.836575 5222 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:07.791999 5257 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:07.841921 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:39:07.842892 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 411992us
I20260927 19:39:07.857944 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:07.856449 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.863874 2037 leader_election.cc:302] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.871245 2142 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260927 19:39:07.878965 2683 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:39:07.881985 1962 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:07.913540 5255 raft_consensus.cc:2726] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:07.917037 5120 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260927 19:39:07.971843 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 55564us
I20260927 19:39:07.904055 4864 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 19:39:07.994706 5265 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260927 19:39:07.942365 5243 raft_consensus.cc:2726] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260927 19:39:07.894021 5133 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.125s
W20260927 19:39:08.030740 2110 leader_election.cc:334] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.527s (SENT)
W20260927 19:39:08.036074 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 172282us
I20260927 19:39:08.037596 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.040172 2037 leader_election.cc:302] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.041639 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.042647 1989 raft_consensus.cc:2379] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 9: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:08.047250 2037 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.051750 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.042542 5222 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 19:39:08.053294 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 304778us
I20260927 19:39:08.044756 5243 raft_consensus.cc:2726] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:08.052116 5101 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.059394 5222 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.043398 5256 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.052354 5233 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260927 19:39:08.055764 2037 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.067216 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
W20260927 19:39:08.054411 5282 log.cc:924] Time spent T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.057s user 0.002s sys 0.000s
I20260927 19:39:08.073913 2683 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 10
I20260927 19:39:08.064105 5120 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:39:08.065308 5158 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260927 19:39:08.067193 5074 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.083289 2037 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:08.086541 2140 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:08.078599 5243 raft_consensus.cc:2726] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.095743 2142 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:08.069608 5124 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:08.094173 5248 consensus_meta.cc:287] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4: Time spent flushing consensus metadata: real 0.550s user 0.011s sys 0.002s
I20260927 19:39:08.075390 5256 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.168216 5248 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.169493 2687 raft_consensus.cc:1226] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:08.170241 5256 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.172410 5248 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.174465 5123 consensus_queue.cc:1046] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260927 19:39:08.115118 5273 log.cc:924] Time spent T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260927 19:39:08.132313 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.189011 2136 raft_consensus.cc:2354] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 9. Current term is 10.
I20260927 19:39:08.085922 5074 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.080569 5102 consensus_queue.cc:1035] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
W20260927 19:39:08.160609 2110 leader_election.cc:334] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515): Timed out: RequestConsensusVote RPC to 127.0.204.130:45515 timed out after 1.607s (SENT)
I20260927 19:39:08.193579 2110 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:08.088999 5120 raft_consensus.cc:2741] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Leader election decision vote started in defunct term 8: won
I20260927 19:39:08.166318 5222 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:08.181500 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 88916.5us
I20260927 19:39:08.199003 1961 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.177062 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.173115 5124 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:08.189322 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:08.202310 1995 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:08.200562 2687 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:08.200717 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.197454 2683 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 10.
I20260927 19:39:08.203946 2136 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:08.206005 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.208047 5063 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:39:08.210636 2137 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
W20260927 19:39:08.211520 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 50961.9us
I20260927 19:39:08.214052 5109 raft_consensus.cc:2726] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260927 19:39:08.214841 5275 log.cc:924] Time spent T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.177s user 0.000s sys 0.003s
I20260927 19:39:08.216531 5074 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:08.219038 2137 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260927 19:39:08.219691 5264 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:08.219668 5271 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:08.221089 5278 consensus_queue.cc:1035] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:08.221788 5243 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:08.241731 5243 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.243688 5243 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.244899 1987 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260927 19:39:08.222213 5120 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:08.245807 5271 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.246002 5120 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.246548 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:08.247125 1987 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:08.247792 5271 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.222743 5244 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:08.248574 5244 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.249294 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.250705 5244 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:08.224900 5283 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
W20260927 19:39:08.217211 5269 log.cc:924] Time spent T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.404s user 0.002s sys 0.000s
I20260927 19:39:08.190078 5123 raft_consensus.cc:3032] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:08.227706 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.230022 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.230526 5287 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:08.230700 5255 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:08.231863 1995 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 10.
I20260927 19:39:08.262611 2062 raft_consensus.cc:1261] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260927 19:39:08.266037 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:08.268644 2065 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260927 19:39:08.238358 2654 leader_election.cc:398] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 9. Current term is 10.
I20260927 19:39:08.239658 5248 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:08.248517 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.256119 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:08.269975 2654 leader_election.cc:401] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:08.230715 1961 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.259611 2064 raft_consensus.cc:1261] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:08.270574 2071 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 7.
I20260927 19:39:08.265116 1987 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:08.272586 2682 raft_consensus.cc:2354] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9.
I20260927 19:39:08.269065 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:08.248047 5120 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.274765 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:08.277621 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 72581.1us
I20260927 19:39:08.278077 2067 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 4.
I20260927 19:39:08.282220 2034 leader_election.cc:398] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9.
I20260927 19:39:08.277168 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:08.282560 2034 leader_election.cc:401] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260927 19:39:08.282938 2063 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 9
W20260927 19:39:08.284206 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45973.8us
I20260927 19:39:08.285202 2108 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:08.260527 5244 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.291647 1992 raft_consensus.cc:1261] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260927 19:39:08.298167 1959 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:08.276510 2037 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:08.306689 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:08.308766 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 78130us
I20260927 19:39:08.308841 5259 raft_consensus.cc:2726] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:08.296339 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 237) took 3309 ms. Trace:
I20260927 19:39:08.261915 5123 raft_consensus.cc:729] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:08.305281 1893 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 6 to 9, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:08.311980 2140 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260927 19:39:08.270016 5233 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.318895 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.326623 2137 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:39:08.311483 1893 catalog_manager.cc:5495] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 5 to 8, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:08.330081 1893 catalog_manager.cc:5495] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.336638 2654 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:08.339335 2037 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:08.341941 2037 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:08.289462 5281 raft_consensus.cc:2726] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:08.299396 5222 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 9. Current term is 10.
I20260927 19:39:08.332435 1893 catalog_manager.cc:5495] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.309970 1890 rpcz_store.cc:276] 0927 19:39:04.974868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:04.984733 (+ 9865us) service_pool.cc:227] Handling call
0927 19:39:08.278097 (+3293364us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:07.209506 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:07.358109 (+148603us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:07.364440 (+ 6331us) write_op.cc:435] Acquired schema lock
0927 19:39:07.367038 (+ 2598us) tablet.cc:660] Decoding operations
0927 19:39:07.516364 (+149326us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:07.516451 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:07.520938 (+ 4487us) tablet.cc:683] Acquiring locks for 9 operations
0927 19:39:07.524957 (+ 4019us) tablet.cc:699] Row locks acquired
0927 19:39:07.524981 (+ 24us) write_op.cc:260] PREPARE: finished
0927 19:39:07.525194 (+ 213us) write_op.cc:270] Start()
0927 19:39:07.525323 (+ 129us) write_op.cc:276] Timestamp: P: 1790537947525140 usec, L: 0
0927 19:39:07.525388 (+ 65us) op_driver.cc:331] REPLICATION: starting
0927 19:39:07.526150 (+ 762us) log.cc:842] Serialized 3699 byte log entry
0927 19:39:07.780789 (+254639us) op_driver.cc:444] REPLICATION: finished
0927 19:39:08.072548 (+291759us) write_op.cc:301] APPLY: starting
0927 19:39:08.072691 (+ 143us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:08.094773 (+ 22082us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:08.094928 (+ 155us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:08.096064 (+ 1136us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:08.102385 (+ 6321us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0927 19:39:08.102438 (+ 53us) write_op.cc:312] APPLY: finished
0927 19:39:08.114341 (+ 11903us) log.cc:842] Serialized 71 byte log entry
0927 19:39:08.173548 (+ 59207us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:08.173682 (+ 134us) write_op.cc:454] Released schema lock
0927 19:39:08.173737 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:08.174024 (+ 287us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:08.174051 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":223177,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":287398,"apply.run_cpu_time_us":9799,"apply.run_wall_time_us":101890,"num_ops":9,"prepare.queue_time_us":192,"prepare.run_cpu_time_us":6141,"prepare.run_wall_time_us":325943,"replication_time_us":255249,"thread_start_us":228923,"threads_started":1}]]}
I20260927 19:39:08.309590 5120 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.354440 5120 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.351827 1893 catalog_manager.cc:5495] T f30da008e35141e99c8237c84d8e66b8 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.356529 5120 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.356328 1893 catalog_manager.cc:5495] T 32bc1524f46f4197b6ac8d29ea44ab77 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 7 to 9, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 9 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.357642 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.361251 1893 catalog_manager.cc:5495] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 3 to 4, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 4 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:08.317687 5158 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:08.318369 5223 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:08.357084 5120 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.381696 2063 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:08.383023 1993 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:08.316701 5264 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 19:39:08.386996 1893 catalog_manager.cc:5495] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 6 to 8, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 8 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.392302 1993 raft_consensus.cc:595] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:08.392529 1961 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.392738 1993 raft_consensus.cc:603] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:08.341426 5300 raft_consensus.cc:2726] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:08.270565 5255 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.391763 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48570 (request call id 1828) took 1113 ms. Trace:
I20260927 19:39:08.393894 1893 catalog_manager.cc:5495] T d9ad83fb0df9449f9081439026d2964f P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.317544 5081 consensus_queue.cc:1035] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260927 19:39:08.384260 5248 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:08.414028 5248 raft_consensus.cc:686] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:08.414989 5248 consensus_queue.cc:237] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.389938 5281 raft_consensus.cc:2726] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:08.397207 1893 catalog_manager.cc:5495] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 5, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 5 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.391863 5120 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.404906 5279 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:08.405339 5263 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260927 19:39:08.343794 5233 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.430311 5233 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.316920 5244 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.432197 5244 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:08.392385 5251 log.cc:924] Time spent T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260927 19:39:08.393190 5243 raft_consensus.cc:2726] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629) with higher term. Message: Invalid argument: T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9.
I20260927 19:39:08.401849 5123 consensus_queue.cc:260] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.435079 5123 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:08.435874 5255 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.391688 5223 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.324474 5280 consensus_queue.cc:1035] T acb3a2a0b9b047249cd5a7535a946fe9 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260927 19:39:08.441437 5223 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.396121 2066 rpcz_store.cc:276] 0927 19:39:07.273183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:07.328062 (+ 54879us) service_pool.cc:227] Handling call
0927 19:39:07.338709 (+ 10647us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:07.339025 (+ 316us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:07.339053 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:07.342292 (+ 3239us) log.cc:842] Serialized 26 byte log entry
0927 19:39:07.570474 (+228182us) raft_consensus.cc:1631] Marking committed up to 6
0927 19:39:07.972105 (+401631us) log.cc:842] Serialized 14 byte log entry
0927 19:39:07.972398 (+ 293us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:07.972474 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:08.216613 (+244139us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:08.378241 (+161628us) raft_consensus.cc:1671] finished
0927 19:39:08.386264 (+ 8023us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:08.386388 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":190927,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":5650,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":27288,"spinlock_wait_cycles":66380032,"thread_start_us":423362,"threads_started":2,"wal-append.queue_time_us":411778}
I20260927 19:39:08.421300 5264 raft_consensus.cc:686] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [term 10 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:08.443323 5264 consensus_queue.cc:237] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.422611 1893 catalog_manager.cc:5495] T 4667efc6973d414fbff5011f6ceaa9b1 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 7 to 9, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 9 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.449081 1893 catalog_manager.cc:5495] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 3 to 4, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:08.453167 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.466085 2687 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:39:08.450560 1893 catalog_manager.cc:5495] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:08.470882 1893 catalog_manager.cc:5495] T 3acf4c2d78fb41ad974812ebf41d0478 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 9 to 10, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 10 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260927 19:39:08.472918 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 132017us
I20260927 19:39:08.473505 1893 catalog_manager.cc:5495] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 7 to 8, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 8 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:08.475186 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.476070 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:08.485706 2062 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
W20260927 19:39:08.468555 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 120925us
I20260927 19:39:08.424378 5279 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:08.490212 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.495715 2064 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260927 19:39:08.497483 2137 raft_consensus.cc:2379] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 in current term 8: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:08.499226 2068 raft_consensus.cc:1261] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260927 19:39:08.503871 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:08.504757 2069 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:08.510707 2654 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:08.509181 5244 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:08.514042 5244 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.516279 5244 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.517416 5274 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.517970 5274 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.525009 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.529687 1991 raft_consensus.cc:2354] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9.
I20260927 19:39:08.531246 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.501268 5279 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:08.533286 1993 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:39:08.530436 2110 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:08.534584 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.535008 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.474452 5255 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260927 19:39:08.545042 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.547355 5074 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:08.547358 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.530977 5263 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:08.550027 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.520280 5274 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.551710 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.509855 5223 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260927 19:39:08.556425 5120 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.553951 5263 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:08.487550 5270 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260927 19:39:08.613590 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.548882 5074 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:39:08.620726 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 83911.4us
I20260927 19:39:08.643074 5271 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.643666 5271 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.646867 5271 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.682194 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.614648 5263 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:08.684120 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.623533 5137 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.692286 2037 leader_election.cc:302] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.629480 5074 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:08.690961 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
W20260927 19:39:08.705293 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 172320us
I20260927 19:39:08.703508 2037 leader_election.cc:302] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.707487 2037 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.709507 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:08.710165 2071 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 8.
I20260927 19:39:08.717839 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.702637 5271 raft_consensus.cc:2726] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.729554 5305 consensus_queue.cc:1035] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260927 19:39:08.732371 5315 consensus_queue.cc:1035] T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260927 19:39:08.744583 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.746212 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.748773 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.713652 5157 raft_consensus.cc:2726] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:08.757751 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:08.758808 2145 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:08.721959 5158 raft_consensus.cc:2726] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:08.703860 5137 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.742770 5300 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20260927 19:39:08.768186 5287 consensus_meta.cc:287] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.503s user 0.011s sys 0.000s
I20260927 19:39:08.773941 5287 raft_consensus.cc:2726] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260927 19:39:08.775928 5314 log.cc:924] Time spent T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
I20260927 19:39:08.779177 5259 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:08.779878 5259 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:08.782267 5254 consensus_queue.cc:1035] T 1b0e603e501b4a8e9e2fe63d3784d744 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:08.780717 5259 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.834103 5317 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.834661 5317 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.835345 5263 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.836819 5317 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:08.839044 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.839797 5258 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:08.840427 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.840305 5258 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:08.841840 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.858604 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.859582 1989 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 5.
I20260927 19:39:08.801453 5265 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.863021 2654 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
W20260927 19:39:08.850068 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 142230us
I20260927 19:39:08.864208 1996 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:08.813133 5281 raft_consensus.cc:2726] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260927 19:39:08.864868 1996 raft_consensus.cc:595] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:39:08.870602 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 150153us
I20260927 19:39:08.873253 2037 leader_election.cc:398] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 8. Current term is 9.
I20260927 19:39:08.875658 2037 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:08.877187 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.882054 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.883036 5233 raft_consensus.cc:2726] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.882838 5315 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 19:39:08.883493 5248 consensus_queue.cc:1035] T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 19:39:08.885403 2037 leader_election.cc:302] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.886199 5323 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:08.886303 5233 raft_consensus.cc:2726] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.886739 5323 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:08.887109 5323 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:08.890914 5258 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:08.892041 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.898469 5137 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:08.881310 5305 raft_consensus.cc:2726] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.903733 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:08.904553 2653 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.904549 1989 raft_consensus.cc:595] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 LEADER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.909847 2653 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.913501 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:08.922528 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:08.923460 2144 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:08.923573 2037 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.928978 2037 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:08.932475 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:08.933458 2683 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:08.881206 5158 raft_consensus.cc:2726] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:08.938797 2037 leader_election.cc:302] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.917230 5248 raft_consensus.cc:2726] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.948537 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:08.950004 1987 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:08.955993 1987 raft_consensus.cc:595] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:08.871358 5265 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:08.925503 5081 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.941860 5233 raft_consensus.cc:2726] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260927 19:39:08.934588 5263 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.947891 5255 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:08.951381 5274 raft_consensus.cc:2726] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:08.973863 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:08.957028 5158 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:08.969944 2071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:08.947443 5157 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:08.969514 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:08.974632 2071 raft_consensus.cc:3037] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 6
I20260927 19:39:08.973650 5255 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260927 19:39:08.978020 5313 log.cc:924] Time spent T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260927 19:39:08.979691 5265 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:08.960505 5244 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.962806 5137 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:08.998051 1995 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:09.015357 1995 raft_consensus.cc:595] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:08.998926 5255 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:08.969105 5081 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.013432 5244 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: NON_VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: true } }
I20260927 19:39:09.005208 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.011093 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.092212 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:39:09.102247 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:09.103024 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:09.108529 2144 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:09.110189 2654 leader_election.cc:302] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:09.092675 2071 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:08.971061 5270 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:08.977640 5325 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:09.012547 5157 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:09.172279 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 45659.9us
I20260927 19:39:09.114362 5248 raft_consensus.cc:2781] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:09.115445 5081 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:09.021548 5137 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.126585 5244 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:09.129206 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:09.171337 5325 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:09.193112 2066 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Committing config change with OpId 8.9: config changed from index 6 to 9, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:09.187881 5270 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:09.236269 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 235355us
I20260927 19:39:09.203719 5244 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:09.277841 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.210120 5325 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:09.233625 5248 raft_consensus.cc:686] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [term 6 LEADER]: Becoming Leader. State: Replica: 4e0335d1a85f421380029b95a91bf0c4, State: Running, Role: LEADER
I20260927 19:39:09.285319 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
W20260927 19:39:09.285539 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 49124.5us
I20260927 19:39:09.292812 5248 consensus_queue.cc:237] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.294876 5157 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.297585 5137 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:39:09.302026 5211 log.cc:924] Time spent T 00000000000000000000000000000000 P 0a0ff9812dcb4ed79a1d0e8f390f6cb0: Append to log took a long time: real 0.273s user 0.000s sys 0.002s
W20260927 19:39:09.253330 5279 consensus_meta.cc:287] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Time spent flushing consensus metadata: real 0.705s user 0.010s sys 0.000s
W20260927 19:39:09.304214 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 49192.6us
I20260927 19:39:09.308593 1962 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:09.307301 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:09.309875 2687 raft_consensus.cc:2379] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 7: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
W20260927 19:39:09.303223 5074 consensus_meta.cc:287] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.593s user 0.006s sys 0.005s
I20260927 19:39:09.312176 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:09.312834 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:09.313810 2068 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:09.331918 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:09.334996 2110 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:09.335806 2037 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:09.337942 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 51524.8us
I20260927 19:39:09.332653 2068 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
W20260927 19:39:09.343463 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 154216us
I20260927 19:39:09.348508 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.350708 5284 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:09.351184 5284 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:09.351594 5284 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:09.353009 5287 raft_consensus.cc:979] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:09.353530 5287 raft_consensus.cc:1067] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:09.354746 5099 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:09.358366 5270 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:09.364459 5254 consensus_queue.cc:1035] T 9e017d73273c4a83bcfdb4c95caa32df P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:09.368468 4979 raft_consensus.cc:979] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:09.368630 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:09.368978 4979 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:09.369488 5257 raft_consensus.cc:1067] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:09.322110 5279 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:09.369071 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 60577us
I20260927 19:39:09.370963 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:09.372170 5279 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.373625 5265 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:09.376535 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:09.377166 2137 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:09.349229 2143 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:09.389099 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:09.389078 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:09.389082 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.390328 2070 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:09.393904 2068 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
W20260927 19:39:09.395038 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 50564.7us
I20260927 19:39:09.394642 2068 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:09.396966 2068 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:09.398175 2137 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:09.404448 2063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:09.405340 2063 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:09.405824 2063 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:09.409454 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:09.409452 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:09.410888 2682 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:09.310125 5244 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.383977 5323 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.323010 5074 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.416356 2070 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:09.429651 5306 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:09.430145 5306 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:09.430506 5306 raft_consensus.cc:3037] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:09.431488 2068 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:09.413862 5338 log.cc:924] Time spent T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.094s user 0.004s sys 0.000s
I20260927 19:39:09.438221 2069 raft_consensus.cc:1261] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:09.438122 2037 leader_election.cc:302] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:09.441411 2110 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:09.438640 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:09.442065 1995 raft_consensus.cc:3032] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:09.439090 2068 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.442330 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:09.442886 2687 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:09.442356 1995 raft_consensus.cc:729] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:09.444494 2136 raft_consensus.cc:1226] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 9. Current term is 10. Ops: []
I20260927 19:39:09.450954 2683 raft_consensus.cc:1226] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260927 19:39:09.453296 2682 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:09.449343 2063 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.462241 5303 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:09.470064 5345 consensus_queue.cc:1035] T 29707548f67846318643f37690b25103 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260927 19:39:09.499568 5303 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:09.500007 5303 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:39:09.503283 5333 log.cc:924] Time spent T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.313s user 0.001s sys 0.001s
W20260927 19:39:09.505443 5334 log.cc:924] Time spent T 8814088a5337420ab29ac2b4b4e0f314 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.214s user 0.002s sys 0.000s
I20260927 19:39:09.507265 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:39:09.509112 5341 log.cc:924] Time spent T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.210s user 0.001s sys 0.001s
I20260927 19:39:09.510097 5323 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:09.440404 5328 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:09.495759 5256 raft_consensus.cc:979] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:39:09.448788 1995 consensus_queue.cc:260] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.523726 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:09.444103 5342 raft_consensus.cc:2726] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:09.489987 5255 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:09.514468 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 50233.6us
W20260927 19:39:09.510404 5339 log.cc:924] Time spent T 85f45e6282f545f19d655ea989e9e123 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20260927 19:39:09.524410 2682 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:09.532492 1959 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:09.526132 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:09.524497 2063 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.528314 1995 raft_consensus.cc:3037] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:09.531052 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:09.534780 1987 raft_consensus.cc:3032] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:09.535192 1987 raft_consensus.cc:729] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:09.536105 1987 consensus_queue.cc:260] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.539026 2653 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:09.552522 1987 raft_consensus.cc:3037] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:09.566969 2682 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:09.568106 2680 raft_consensus.cc:1226] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:09.573990 2034 leader_election.cc:302] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:09.574419 2063 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:09.533754 4864 raft_consensus.cc:1067] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 LEADER]: Signalling peer 69f09866d0a1477585f05b2f82ff4c67 to start an election
I20260927 19:39:09.576768 5279 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260927 19:39:09.590399 5303 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.526775 5328 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.578814 5342 raft_consensus.cc:2781] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:09.571949 5248 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260927 19:39:09.592504 5335 log.cc:924] Time spent T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.302s user 0.000s sys 0.001s
I20260927 19:39:09.603045 5328 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.542950 5256 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:09.601881 5244 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:09.633860 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:09.625100 1995 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:09.611052 2687 raft_consensus.cc:2454] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:09.628913 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:09.629820 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:09.630821 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:09.633833 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 168157us
I20260927 19:39:09.635435 2683 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260927 19:39:09.602401 5074 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:39:09.597302 5343 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
I20260927 19:39:09.640368 2034 leader_election.cc:302] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:09.524986 5325 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.633970 5233 raft_consensus.cc:2726] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:09.527220 5306 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.643392 5342 raft_consensus.cc:686] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [term 6 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:09.658243 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.643677 5256 raft_consensus.cc:686] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:39:09.666090 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 207137us
I20260927 19:39:09.674309 2110 leader_election.cc:302] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:09.639698 5271 consensus_queue.cc:1046] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260927 19:39:09.653890 5074 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:09.675369 1987 raft_consensus.cc:2454] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:09.688094 5255 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.667246 5292 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260927 19:39:09.699970 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:09.700488 2683 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:09.673373 5305 raft_consensus.cc:2781] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:09.679379 5299 raft_consensus.cc:2726] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260927 19:39:09.732816 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 60166us
I20260927 19:39:09.735499 2683 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:09.738493 2034 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:09.740589 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:09.708617 5133 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:09.744849 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:09.745499 2136 raft_consensus.cc:3037] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
W20260927 19:39:09.749195 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 65779.4us
I20260927 19:39:09.753636 2136 raft_consensus.cc:2454] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:09.757324 2144 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:40164
I20260927 19:39:09.757997 2144 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:09.758358 2144 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 11 FOLLOWER]: Advancing to term 12
I20260927 19:39:09.758296 2111 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:09.735942 5342 consensus_queue.cc:237] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.739856 5305 raft_consensus.cc:686] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 8 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:09.709461 5074 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.749024 5300 raft_consensus.cc:2726] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:09.750211 5270 raft_consensus.cc:2781] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:09.800104 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:09.802861 2067 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:09.802038 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:09.772312 5303 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:09.773603 5299 raft_consensus.cc:2726] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:09.811728 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:09.813897 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 55197.1us
I20260927 19:39:09.751572 5133 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.820897 2138 raft_consensus.cc:2454] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:09.815604 2144 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.822775 2037 leader_election.cc:302] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:09.833659 2144 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 12 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.836030 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:09.836812 2683 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 11 FOLLOWER]: Advancing to term 12
I20260927 19:39:09.769188 5256 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.847342 5359 consensus_queue.cc:1046] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20260927 19:39:09.848886 2683 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 12.
I20260927 19:39:09.852135 2108 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:09.853178 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:09.854359 2683 raft_consensus.cc:2379] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 7: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:09.862294 5266 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:09.862941 5266 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.864912 5266 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:09.820147 5270 raft_consensus.cc:686] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:09.867372 5270 consensus_queue.cc:237] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:09.869920 5360 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260927 19:39:09.841863 5325 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:09.871215 1992 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56618 (request call id 593) took 1369 ms. Trace:
I20260927 19:39:09.871620 1992 rpcz_store.cc:276] 0927 19:39:08.502059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:08.502275 (+ 216us) service_pool.cc:227] Handling call
0927 19:39:08.502565 (+ 290us) raft_consensus.cc:1457] Updating replica for 1 ops
0927 19:39:08.812702 (+310137us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:08.812733 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0927 19:39:09.383899 (+571166us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b2c00028668. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f7912345e11 00007f7912355442 00007f791399dc11 00007f791396258a 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772
0927 19:39:09.384306 (+ 407us) log.cc:842] Serialized 26 byte log entry
0927 19:39:09.638162 (+253856us) raft_consensus.cc:1631] Marking committed up to 9
0927 19:39:09.638204 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:09.638295 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:09.870849 (+232554us) raft_consensus.cc:1671] finished
0927 19:39:09.870872 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:09.871130 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":289021,"raft.queue_time_us":308943,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":2678,"spinlock_wait_cycles":2338560,"thread_start_us":253570,"threads_started":1,"wal-append.queue_time_us":297861}
W20260927 19:39:09.874423 5363 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260927 19:39:09.854359 5306 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:09.878999 5305 consensus_queue.cc:237] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:09.835417 5300 raft_consensus.cc:2781] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:09.883422 5133 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:09.884539 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:09.886632 5074 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
W20260927 19:39:09.886986 5340 log.cc:924] Time spent T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.582s user 0.004s sys 0.000s
I20260927 19:39:09.887055 5265 raft_consensus.cc:3032] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Stepping down as leader of term 9
I20260927 19:39:09.887576 5265 raft_consensus.cc:729] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:09.888362 5265 consensus_queue.cc:260] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.894276 5081 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 12 FOLLOWER]: Leader election won for term 12
I20260927 19:39:09.894920 5081 raft_consensus.cc:686] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [term 12 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:09.895848 5081 consensus_queue.cc:237] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:09.897411 5284 consensus_meta.cc:287] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67: Time spent flushing consensus metadata: real 0.545s user 0.012s sys 0.000s
I20260927 19:39:09.899487 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:09.899923 2067 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:09.906827 2063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:09.905762 2062 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:09.907096 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:09.897830 5284 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.916852 2136 raft_consensus.cc:1226] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 8. Current term is 9. Ops: []
I20260927 19:39:09.919211 5263 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:09.905762 5300 raft_consensus.cc:686] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:09.917986 5265 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:09.938791 2110 leader_election.cc:302] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:09.943152 2108 leader_election.cc:302] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:09.965148 5257 raft_consensus.cc:979] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:09.965812 5257 raft_consensus.cc:1067] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:09.956877 5264 raft_consensus.cc:2726] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:09.948858 5074 raft_consensus.cc:2781] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election won for term 7
W20260927 19:39:09.981231 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 220592us
I20260927 19:39:09.984165 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:09.984846 1992 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:09.985844 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:09.986629 2067 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:09.988380 2063 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:09.988961 2063 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:09.989387 2063 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:09.994374 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:09.993167 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:09.972257 5327 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:09.998479 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:10.001721 1993 raft_consensus.cc:3037] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:10.010087 2063 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:09.988698 5074 raft_consensus.cc:686] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:10.026288 1993 raft_consensus.cc:2454] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:10.024618 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:10.030664 2138 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:10.023180 2063 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:10.033442 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:10.034304 2139 raft_consensus.cc:2354] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 8. Current term is 9.
I20260927 19:39:10.039619 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:10.040419 1988 raft_consensus.cc:3032] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:10.040750 1988 raft_consensus.cc:729] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:10.048552 1961 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:10.044333 1988 consensus_queue.cc:260] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.054986 2138 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:10.056766 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:10.058672 2037 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:10.022616 5376 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:10.061650 2654 leader_election.cc:398] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 8. Current term is 9.
I20260927 19:39:10.062579 2654 leader_election.cc:401] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260927 19:39:10.063925 5258 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:10.064378 5258 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:10.064755 5258 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:09.980738 5300 consensus_queue.cc:237] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:10.073832 5074 consensus_queue.cc:237] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.078943 5243 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:10.079576 5243 raft_consensus.cc:686] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:10.080475 5243 consensus_queue.cc:237] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.055785 5284 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:10.085217 2138 raft_consensus.cc:1226] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 8. Current term is 9. Ops: []
I20260927 19:39:10.085558 5376 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:10.087249 5332 consensus_queue.cc:1046] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:39:10.087852 5376 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:10.093286 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:10.012753 5327 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
W20260927 19:39:10.113797 1961 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 57952.2us
I20260927 19:39:10.121814 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:10.120942 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:10.124723 1993 raft_consensus.cc:3032] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 LEADER]: Stepping down as leader of term 11
I20260927 19:39:10.125142 1993 raft_consensus.cc:729] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:10.132236 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:10.133999 1989 raft_consensus.cc:1261] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260927 19:39:10.135795 2111 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
W20260927 19:39:10.133291 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 71845us
I20260927 19:39:10.142956 2111 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:10.128361 1993 consensus_queue.cc:260] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:10.158600 2688 raft_consensus.cc:1226] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260927 19:39:10.167186 1990 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:60280 (request call id 1238) took 1843 ms. Trace:
I20260927 19:39:10.168913 1990 rpcz_store.cc:276] 0927 19:39:08.319246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:08.319691 (+ 445us) service_pool.cc:227] Handling call
0927 19:39:10.154238 (+1834547us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400034dd4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79188a7c6e 00007f7911f9e807 00007f7911f9e792 00007f7911f9e6c1 00007f7911f9e63c 00007f7911f9d8f2 00007f7911d8c94c 00007f7911d8bd86 00007f7911d8b72f
0927 19:39:10.162822 (+ 8584us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2421704960}
I20260927 19:39:10.170113 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:10.145450 5266 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:10.167587 5292 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260927 19:39:10.203253 5302 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045) with higher term. Message: Invalid argument: T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for earlier term 8. Current term is 9.
W20260927 19:39:10.205168 5383 log.cc:924] Time spent T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260927 19:39:10.196259 5299 consensus_queue.cc:1035] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.010s
I20260927 19:39:10.178050 5255 consensus_queue.cc:1046] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260927 19:39:10.210592 5281 raft_consensus.cc:3032] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Stepping down as leader of term 5
I20260927 19:39:10.210983 5281 raft_consensus.cc:729] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:10.211063 2683 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:10.214522 5327 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:10.225330 5244 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260927 19:39:10.228884 5258 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.230979 5258 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:10.233155 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:10.233727 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:10.234511 1992 raft_consensus.cc:2454] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 10.
I20260927 19:39:10.234571 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:10.238143 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:10.241541 1988 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:10.247849 1962 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:10.252245 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 76357.8us
I20260927 19:39:10.262841 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:10.287871 2142 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:10.267731 5387 raft_consensus.cc:2726] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260927 19:39:10.276604 5244 raft_consensus.cc:2726] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:10.309610 5120 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
W20260927 19:39:10.322641 5355 log.cc:924] Time spent T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.549s user 0.007s sys 0.000s
I20260927 19:39:10.328910 1988 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
W20260927 19:39:10.345970 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 176744us
I20260927 19:39:10.351619 2136 raft_consensus.cc:1261] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:10.352523 2070 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:10.353796 2108 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:10.362071 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 197177us
I20260927 19:39:10.371997 5281 consensus_queue.cc:260] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:10.357654 5254 consensus_queue.cc:1046] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260927 19:39:10.383920 1988 raft_consensus.cc:1261] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260927 19:39:10.387260 5133 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:10.393635 2687 raft_consensus.cc:1261] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:10.404919 5158 consensus_queue.cc:1035] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260927 19:39:10.398842 5254 raft_consensus.cc:3032] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:10.419698 5281 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:10.429615 1994 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:10.406028 5244 consensus_queue.cc:1035] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260927 19:39:10.441488 5362 raft_consensus.cc:2726] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:10.262832 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:10.419059 5254 raft_consensus.cc:729] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:10.455787 5254 consensus_queue.cc:260] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.418035 5323 consensus_queue.cc:1035] T 876ffc23594245239e2688f5992774cf P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260927 19:39:10.468034 2066 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60518 (request call id 483) took 1492 ms. Trace:
I20260927 19:39:10.468600 2066 rpcz_store.cc:276] 0927 19:39:08.975415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:08.975564 (+ 149us) service_pool.cc:227] Handling call
0927 19:39:08.998671 (+ 23107us) raft_consensus.cc:1457] Updating replica for 2 ops
0927 19:39:09.130965 (+132294us) raft_consensus.cc:1520] Early marking committed up to index 0
0927 19:39:09.130994 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0927 19:39:09.132110 (+ 1116us) log.cc:842] Serialized 590 byte log entry
0927 19:39:09.192474 (+ 60364us) raft_consensus.cc:1631] Marking committed up to 9
0927 19:39:09.581184 (+388710us) log.cc:842] Serialized 14 byte log entry
0927 19:39:09.596303 (+ 15119us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b44000232cc. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f791087ef11 00007f791094ca95 00007f791094d4b2 00007f79135f5511 00007f79135f0e1a 00007f79135e4bee 00007f79135e3f36 00007f79135e3b58 00007f79135ed551 00007f79135ed509 00007f79135ed499
0927 19:39:09.628085 (+ 31782us) raft_consensus.cc:1683] Filling consensus response to leader.
0927 19:39:10.467653 (+839568us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400324954. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7471 00007f7913962f51 00007f791396199e 00007f79188a32e9 00007f7911f89684 00007f7911f89612 00007f7911f89541 00007f7911f894bc 00007f7911f88772 00007f7911d8c94c 00007f7911d8bd86
0927 19:39:10.467729 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0927 19:39:10.467785 (+ 56us) raft_consensus.cc:1671] finished
0927 19:39:10.467801 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0927 19:39:10.467963 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64681,"raft.queue_time_us":86843,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":16952,"spinlock_wait_cycles":328171776,"thread_start_us":59969,"threads_started":1,"wal-append.queue_time_us":107886}
I20260927 19:39:10.475320 1892 catalog_manager.cc:5495] T 228a1cf257434cbe9e5616dbd94e1864 P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: config changed from index 4 to 8, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 8 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:10.476537 5254 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:10.477524 1993 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 11 FOLLOWER]: Advancing to term 12
I20260927 19:39:10.477715 2680 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:10.481702 1987 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260927 19:39:10.483248 5244 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260927 19:39:10.480943 5256 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:10.492492 1993 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 12.
I20260927 19:39:10.495253 2138 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260927 19:39:10.500810 5256 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:10.508114 1892 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 reported cstate change: term changed from 7 to 8, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 8 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:10.531729 2683 raft_consensus.cc:1261] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260927 19:39:10.533318 5244 consensus_queue.cc:1035] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260927 19:39:10.556876 2111 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:39:10.573720 5298 raft_consensus.cc:2781] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Leader election won for term 10
I20260927 19:39:10.574294 5298 raft_consensus.cc:686] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [term 10 LEADER]: Becoming Leader. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:10.575212 5298 consensus_queue.cc:237] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.586301 5081 consensus_queue.cc:1046] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:10.587677 5264 raft_consensus.cc:3032] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:10.588086 5264 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:10.588850 5264 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.592615 2688 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 19:39:10.594089 5271 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:10.600862 1993 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260927 19:39:10.611652 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 243) took 3790 ms. Trace:
I20260927 19:39:10.612669 1893 rpcz_store.cc:276] 0927 19:39:06.820325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:06.820810 (+ 485us) service_pool.cc:227] Handling call
0927 19:39:10.610785 (+3789975us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:08.829490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:08.830080 (+ 590us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:08.830133 (+ 53us) write_op.cc:435] Acquired schema lock
0927 19:39:08.830148 (+ 15us) tablet.cc:660] Decoding operations
0927 19:39:08.831623 (+ 1475us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:08.831687 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:08.831718 (+ 31us) tablet.cc:683] Acquiring locks for 15 operations
0927 19:39:08.832911 (+ 1193us) tablet.cc:699] Row locks acquired
0927 19:39:08.832931 (+ 20us) write_op.cc:260] PREPARE: finished
0927 19:39:08.833073 (+ 142us) write_op.cc:270] Start()
0927 19:39:08.833193 (+ 120us) write_op.cc:276] Timestamp: P: 1790537948833028 usec, L: 0
0927 19:39:08.833215 (+ 22us) op_driver.cc:331] REPLICATION: starting
0927 19:39:08.906355 (+ 73140us) log.cc:842] Serialized 6012 byte log entry
0927 19:39:09.495040 (+588685us) op_driver.cc:444] REPLICATION: finished
0927 19:39:09.711680 (+216640us) write_op.cc:301] APPLY: starting
0927 19:39:09.734863 (+ 23183us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:09.782159 (+ 47296us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:09.838822 (+ 56663us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:09.878476 (+ 39654us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:09.878789 (+ 313us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0927 19:39:09.878846 (+ 57us) write_op.cc:312] APPLY: finished
0927 19:39:09.879237 (+ 391us) log.cc:842] Serialized 107 byte log entry
0927 19:39:09.879414 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:09.879589 (+ 175us) write_op.cc:454] Released schema lock
0927 19:39:09.879645 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:09.879918 (+ 273us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:09.879953 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2824,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":182339,"apply.run_cpu_time_us":5042,"apply.run_wall_time_us":172724,"num_ops":15,"prepare.queue_time_us":192647,"prepare.run_cpu_time_us":5178,"prepare.run_wall_time_us":90692,"replication_time_us":638434,"thread_start_us":243940,"threads_started":2}]]}
I20260927 19:39:10.631253 2070 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 5 to 7, 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:10.628561 5264 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:10.606359 5271 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260927 19:39:10.659843 2066 raft_consensus.cc:1261] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260927 19:39:10.662190 5278 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:10.690125 1992 raft_consensus.cc:1261] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260927 19:39:10.752689 5412 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260927 19:39:10.759393 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:10.763022 1992 raft_consensus.cc:595] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
W20260927 19:39:10.773164 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 50531.7us
I20260927 19:39:10.773283 5081 consensus_queue.cc:1035] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:10.778086 2136 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "794b9dbf053c445393588ff152553db5"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:10.778736 2136 raft_consensus.cc:595] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:10.790658 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:10.791414 1990 raft_consensus.cc:595] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:10.805686 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83f97251c21d4e979bce8419c208369c"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:10.806406 1992 raft_consensus.cc:595] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:10.810658 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "69f09866d0a1477585f05b2f82ff4c67"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:10.811632 1992 raft_consensus.cc:595] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Received request to transfer leadership to TS 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:10.812115 1992 raft_consensus.cc:603] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:10.816741 1990 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:10.817448 1990 raft_consensus.cc:595] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:10.818557 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260927 19:39:10.820206 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:10.820801 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:10.821931 5300 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:39:10.834172 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 117710us
I20260927 19:39:10.839448 2136 raft_consensus.cc:1261] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260927 19:39:10.840502 2143 raft_consensus.cc:1261] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260927 19:39:10.839600 5300 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.851120 5300 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:10.879838 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:10.881239 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:10.915871 5306 consensus_queue.cc:1035] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 19:39:10.898447 5300 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:10.917318 5300 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.917879 5306 consensus_queue.cc:1035] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:39:10.919538 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:10.920261 5306 raft_consensus.cc:2726] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:10.920181 5300 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:10.931293 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:10.934358 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:10.941236 1990 raft_consensus.cc:2379] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 10: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:10.943266 2037 leader_election.cc:302] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cc2df048f424e08803a390946004df0; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:10.952507 5256 raft_consensus.cc:979] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.953424 5398 raft_consensus.cc:979] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.953894 5398 raft_consensus.cc:979] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.954103 5102 raft_consensus.cc:979] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.954532 5081 raft_consensus.cc:1067] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.954802 5398 raft_consensus.cc:1067] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.955724 5398 raft_consensus.cc:1067] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
W20260927 19:39:10.960646 5414 log.cc:924] Time spent T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260927 19:39:10.961264 1994 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260927 19:39:10.963343 2064 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:10.963958 2064 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:10.964282 2064 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:10.965533 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:10.966312 2070 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:10.966748 2070 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:10.978435 2070 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.969761 5281 raft_consensus.cc:3037] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:10.986956 2070 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:10.987888 2067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "794b9dbf053c445393588ff152553db5"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48570
I20260927 19:39:10.973081 5257 raft_consensus.cc:1067] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:10.988524 2067 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:10.988792 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:10.990973 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:10.991437 1988 raft_consensus.cc:3032] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:10.991690 2144 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:10.991927 1988 raft_consensus.cc:729] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:10.990931 2067 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:10.992920 1988 consensus_queue.cc:260] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:10.975445 5328 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260927 19:39:11.019330 1988 raft_consensus.cc:3037] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:11.027726 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:11.028594 2070 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:11.029059 2070 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:11.017037 2067 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.051666 1988 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.051461 2070 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.051808 2144 raft_consensus.cc:2454] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.046685 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:11.057293 1992 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:11.059775 2070 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.061430 2037 leader_election.cc:302] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.033155 2064 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.065068 2064 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.062000 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:11.067595 2144 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:11.068706 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:11.073252 2067 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.070887 1988 raft_consensus.cc:3032] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:11.077791 1988 raft_consensus.cc:729] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:11.078657 1988 consensus_queue.cc:260] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.080094 1988 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:11.072561 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:11.081382 1994 raft_consensus.cc:3032] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:11.081683 1994 raft_consensus.cc:729] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:11.081729 2144 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.100798 1992 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:11.109642 2037 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:11.111245 5300 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:11.111804 5300 raft_consensus.cc:686] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:11.116278 5279 raft_consensus.cc:979] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:11.112732 5300 consensus_queue.cc:237] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.133618 2037 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.135633 5158 raft_consensus.cc:2781] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:11.136125 5158 raft_consensus.cc:686] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:11.142980 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:11.152592 2144 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:11.100550 1994 consensus_queue.cc:260] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.155369 1994 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:11.083221 5271 raft_consensus.cc:2781] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:11.158198 1988 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.158946 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:11.159981 2136 raft_consensus.cc:3032] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Stepping down as leader of term 8
I20260927 19:39:11.160236 2136 raft_consensus.cc:729] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 69f09866d0a1477585f05b2f82ff4c67, State: Running, Role: LEADER
I20260927 19:39:11.148674 4864 raft_consensus.cc:1067] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:11.160905 2136 consensus_queue.cc:260] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.136941 5158 consensus_queue.cc:237] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.220084 2144 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:11.274755 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83f97251c21d4e979bce8419c208369c"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:11.172500 5271 raft_consensus.cc:686] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
W20260927 19:39:11.319000 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 219899us
I20260927 19:39:11.315753 1994 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:11.256656 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 159) took 4382 ms. Trace:
I20260927 19:39:11.351603 2037 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.273535 2136 raft_consensus.cc:3037] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
W20260927 19:39:11.233847 5422 log.cc:924] Time spent T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260927 19:39:11.326202 1891 catalog_manager.cc:5495] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 7 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:11.349772 2070 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:11.356237 2070 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:11.240720 1992 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260927 19:39:11.356812 1891 catalog_manager.cc:5495] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:11.366935 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet 228a1cf257434cbe9e5616dbd94e1864 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40144
I20260927 19:39:11.352795 5271 consensus_queue.cc:237] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.360612 1891 catalog_manager.cc:5495] T 63522eab31e04c8191338bb22acb0297 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:11.362551 5244 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:11.377776 1891 catalog_manager.cc:5495] T b2f6360a6b784953bc89cd72712a5e7d P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 5 to 6, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 6 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:11.350224 1892 rpcz_store.cc:276] 0927 19:39:06.874396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:06.874581 (+ 185us) service_pool.cc:227] Handling call
0927 19:39:11.256581 (+4382000us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:10.651705 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:10.652205 (+ 500us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:10.652238 (+ 33us) write_op.cc:435] Acquired schema lock
0927 19:39:10.652254 (+ 16us) tablet.cc:660] Decoding operations
0927 19:39:10.652643 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:10.652713 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:10.652737 (+ 24us) tablet.cc:683] Acquiring locks for 2 operations
0927 19:39:10.653033 (+ 296us) tablet.cc:699] Row locks acquired
0927 19:39:10.653051 (+ 18us) write_op.cc:260] PREPARE: finished
0927 19:39:10.653194 (+ 143us) write_op.cc:270] Start()
0927 19:39:10.653291 (+ 97us) write_op.cc:276] Timestamp: P: 1790537950653149 usec, L: 0
0927 19:39:10.653313 (+ 22us) op_driver.cc:331] REPLICATION: starting
0927 19:39:10.691141 (+ 37828us) log.cc:842] Serialized 915 byte log entry
0927 19:39:10.951424 (+260283us) op_driver.cc:444] REPLICATION: finished
0927 19:39:11.075188 (+123764us) write_op.cc:301] APPLY: starting
0927 19:39:11.075274 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:11.075377 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:11.075483 (+ 106us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:11.075820 (+ 337us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:11.075986 (+ 166us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0927 19:39:11.076029 (+ 43us) write_op.cc:312] APPLY: finished
0927 19:39:11.076220 (+ 191us) log.cc:842] Serialized 29 byte log entry
0927 19:39:11.076376 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:11.076460 (+ 84us) write_op.cc:454] Released schema lock
0927 19:39:11.076495 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:11.076731 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:11.076756 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23759,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":123474,"apply.run_cpu_time_us":1813,"apply.run_wall_time_us":1813,"num_ops":2,"prepare.queue_time_us":121883,"prepare.run_cpu_time_us":2680,"prepare.run_wall_time_us":48575,"replication_time_us":297987,"thread_start_us":198191,"threads_started":2}]]}
I20260927 19:39:11.394536 2136 raft_consensus.cc:2454] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:11.394260 1891 catalog_manager.cc:5495] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 3 to 4, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 4 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:11.395212 5244 raft_consensus.cc:686] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:11.400774 5244 consensus_queue.cc:237] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.386191 5280 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260927 19:39:11.393826 2070 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.431946 1891 catalog_manager.cc:5495] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:11.445171 2070 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.446414 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:11.448073 2143 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:11.449136 5281 raft_consensus.cc:2726] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:11.459153 2067 raft_consensus.cc:1261] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 10 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260927 19:39:11.456799 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:11.440162 1891 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 5, leader changed from 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:11.471112 1994 raft_consensus.cc:3032] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:11.474311 1994 raft_consensus.cc:729] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:11.475595 1994 consensus_queue.cc:260] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.476615 5298 consensus_queue.cc:1035] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260927 19:39:11.512607 1891 catalog_manager.cc:5495] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
W20260927 19:39:11.516587 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 49936.5us
W20260927 19:39:11.522197 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 91256.4us
I20260927 19:39:11.516954 1994 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:39:11.526468 2654 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.204.190:41277, user_credentials={real_user=slave}} blocked reactor thread for 51004.2us
I20260927 19:39:11.518024 1891 catalog_manager.cc:5495] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 9, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 9 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:11.530532 2687 raft_consensus.cc:1261] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 12 FOLLOWER]: Refusing update from remote peer 69f09866d0a1477585f05b2f82ff4c67: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260927 19:39:11.540500 2143 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.544785 2037 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:11.531186 1891 catalog_manager.cc:5495] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 8, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 8 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:11.557700 1891 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 8 to 9, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132). New cstate: current_term: 9 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } attrs { replace: true } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:11.559918 1891 catalog_manager.cc:5495] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:11.565627 1994 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.565292 1891 catalog_manager.cc:5495] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 8, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } }
I20260927 19:39:11.567282 5243 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:11.571233 5158 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
W20260927 19:39:11.661549 5428 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260927 19:39:11.680025 5158 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.583480 5256 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:11.680893 5120 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:11.680974 5256 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:11.681551 5120 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.682075 5158 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.683233 5256 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:11.683454 5120 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.667210 5243 raft_consensus.cc:686] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:11.685871 5243 consensus_queue.cc:237] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.668337 5101 consensus_queue.cc:1035] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260927 19:39:11.666455 5432 tablet_replica.cc:330] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:39:11.696851 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:11.698014 1988 raft_consensus.cc:2379] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 7: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:11.699033 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:11.699851 2066 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:11.703583 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:11.704392 1988 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:11.705504 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.706537 5331 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:11.706971 5331 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:11.707387 5331 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:11.719449 5331 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:11.722044 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.733580 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:11.741338 2136 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:11.742470 2143 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:11.744320 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:11.745385 2143 raft_consensus.cc:2379] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 9: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
W20260927 19:39:11.760231 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 123416us
I20260927 19:39:11.731722 5424 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:11.761550 5300 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:11.770748 5424 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:11.782459 2143 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260927 19:39:11.780567 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:11.782032 5300 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:11.788491 5300 raft_consensus.cc:3037] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:11.797466 5109 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:11.797976 5109 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.799880 5109 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:11.806880 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:11.807705 1988 raft_consensus.cc:2396] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260927 19:39:11.813486 5432 raft_consensus.cc:2229] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Raft consensus shutting down.
I20260927 19:39:11.814267 5432 raft_consensus.cc:2258] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Raft consensus is shut down!
I20260927 19:39:11.815344 5331 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:11.817265 5432 ts_tablet_manager.cc:1905] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:39:11.827113 5281 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260927 19:39:11.831499 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 50999.4us
I20260927 19:39:11.824039 5300 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.838235 1989 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260927 19:39:11.840392 1989 raft_consensus.cc:1261] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260927 19:39:11.845660 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 193617us
I20260927 19:39:11.847641 5424 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:11.832927 5281 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:11.850160 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:11.851650 1992 raft_consensus.cc:1261] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:11.854756 5281 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:39:11.856714 1990 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.010s sys 0.004s
W20260927 19:39:11.861853 1990 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s user 0.010s sys 0.004s
I20260927 19:39:11.862123 2065 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:11.864491 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:11.864799 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 81887us
I20260927 19:39:11.865484 2065 raft_consensus.cc:2396] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260927 19:39:11.882320 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:11.883112 2680 raft_consensus.cc:2379] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:11.883767 2683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:11.884541 2683 raft_consensus.cc:2379] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d1bc2cd9efe43399ddba396f5ddff85 in current term 6: Already voted for candidate 4e0335d1a85f421380029b95a91bf0c4 in this term.
I20260927 19:39:11.881050 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:11.885730 1994 raft_consensus.cc:2396] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 19:39:11.886389 1961 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.893433 2110 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:11.899510 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:11.902182 2066 raft_consensus.cc:2396] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e0335d1a85f421380029b95a91bf0c4 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260927 19:39:11.904502 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:11.895289 5133 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:11.880023 5441 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260927 19:39:11.908241 5300 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.909758 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:11.895231 5243 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260927 19:39:11.884441 5271 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260927 19:39:11.920606 2654 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:11.899127 5368 raft_consensus.cc:2726] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:11.929440 1994 raft_consensus.cc:2454] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:11.931253 2136 raft_consensus.cc:2379] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 in current term 9: Already voted for candidate 69f09866d0a1477585f05b2f82ff4c67 in this term.
I20260927 19:39:11.932224 2037 leader_election.cc:302] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:11.925799 5327 consensus_queue.cc:1035] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260927 19:39:11.935036 5306 raft_consensus.cc:2781] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader election won for term 9
I20260927 19:39:11.935515 5306 raft_consensus.cc:686] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:11.936694 1990 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260927 19:39:11.936304 5306 consensus_queue.cc:237] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.922917 5244 consensus_queue.cc:1035] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260927 19:39:11.931232 5345 raft_consensus.cc:2726] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:11.946900 5281 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.949028 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:11.923632 5133 raft_consensus.cc:686] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:11.960160 5133 consensus_queue.cc:237] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:11.962657 2143 raft_consensus.cc:1261] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260927 19:39:11.965514 5281 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:11.966351 5441 consensus_queue.cc:1035] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:39:11.975131 4864 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:11.975724 4864 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:11.979718 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:11.980504 2136 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:11.981714 1962 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:11.982549 5373 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:11.982933 5373 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:11.983273 5373 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:11.995565 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:11.977965 4864 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:12.002494 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 71404.3us
I20260927 19:39:12.019275 2144 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:12.022785 2037 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:12.023587 5120 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader election won for term 7
I20260927 19:39:12.023818 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:12.024231 5120 raft_consensus.cc:686] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:12.025345 5120 consensus_queue.cc:237] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:12.029708 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:11.991330 5281 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260927 19:39:12.090699 2144 raft_consensus.cc:1261] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260927 19:39:12.102957 5327 consensus_queue.cc:1035] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260927 19:39:12.143292 5373 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:12.144703 1992 raft_consensus.cc:2354] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:12.143679 5432 ts_tablet_manager.cc:1918] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 8.7
I20260927 19:39:12.187879 5432 log.cc:1196] T 228a1cf257434cbe9e5616dbd94e1864 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/228a1cf257434cbe9e5616dbd94e1864
I20260927 19:39:12.192387 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet 228a1cf257434cbe9e5616dbd94e1864 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
W20260927 19:39:12.146816 5461 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.065s user 0.003s sys 0.000s
I20260927 19:39:12.147233 5373 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:12.156633 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:12.195396 2142 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
W20260927 19:39:12.157202 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 56469.6us
I20260927 19:39:12.159317 2139 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:12.160199 1989 raft_consensus.cc:1261] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260927 19:39:12.173754 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:12.208847 2037 leader_election.cc:398] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Vote denied by peer 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941) with higher term. Message: Invalid argument: T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8.
I20260927 19:39:12.233716 5243 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:12.234553 5441 consensus_queue.cc:1035] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:12.250389 2142 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:12.251868 1962 leader_election.cc:302] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0
I20260927 19:39:12.271494 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:12.322010 1992 raft_consensus.cc:595] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:12.264413 5133 raft_consensus.cc:2781] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election won for term 8
W20260927 19:39:12.328130 5452 log.cc:924] Time spent T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260927 19:39:12.338112 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 238) took 1840 ms. Trace:
I20260927 19:39:12.360033 1891 rpcz_store.cc:276] 0927 19:39:10.424769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:10.424884 (+ 115us) service_pool.cc:227] Handling call
0927 19:39:12.264654 (+1839770us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:11.844070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:11.844637 (+ 567us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:11.844670 (+ 33us) write_op.cc:435] Acquired schema lock
0927 19:39:11.844688 (+ 18us) tablet.cc:660] Decoding operations
0927 19:39:11.845997 (+ 1309us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:11.846080 (+ 83us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:11.846111 (+ 31us) tablet.cc:683] Acquiring locks for 13 operations
0927 19:39:11.847302 (+ 1191us) tablet.cc:699] Row locks acquired
0927 19:39:11.847322 (+ 20us) write_op.cc:260] PREPARE: finished
0927 19:39:11.847491 (+ 169us) write_op.cc:270] Start()
0927 19:39:11.847599 (+ 108us) write_op.cc:276] Timestamp: P: 1790537951847421 usec, L: 0
0927 19:39:11.847630 (+ 31us) op_driver.cc:331] REPLICATION: starting
0927 19:39:11.848342 (+ 712us) log.cc:842] Serialized 5216 byte log entry
0927 19:39:11.990587 (+142245us) op_driver.cc:444] REPLICATION: finished
0927 19:39:12.044967 (+ 54380us) write_op.cc:301] APPLY: starting
0927 19:39:12.045061 (+ 94us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:12.045425 (+ 364us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:12.045558 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:12.179729 (+134171us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:12.179977 (+ 248us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0927 19:39:12.180021 (+ 44us) write_op.cc:312] APPLY: finished
0927 19:39:12.180328 (+ 307us) log.cc:842] Serialized 95 byte log entry
0927 19:39:12.195450 (+ 15122us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:12.195984 (+ 534us) write_op.cc:454] Released schema lock
0927 19:39:12.196061 (+ 77us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:12.198612 (+ 2551us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:12.198646 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":65019,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":54083,"apply.run_cpu_time_us":4445,"apply.run_wall_time_us":156916,"num_ops":13,"prepare.queue_time_us":111303,"prepare.run_cpu_time_us":4536,"prepare.run_wall_time_us":4538,"replication_time_us":142831,"thread_start_us":146707,"threads_started":2}]]}
I20260927 19:39:12.360718 5133 raft_consensus.cc:686] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
W20260927 19:39:12.391834 5455 log.cc:924] Time spent T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260927 19:39:12.423979 5133 consensus_queue.cc:237] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:12.444517 1989 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260927 19:39:12.470058 5459 log.cc:924] Time spent T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
W20260927 19:39:12.491099 5462 log.cc:924] Time spent T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260927 19:39:12.499102 5441 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:12.686620 2144 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "794b9dbf053c445393588ff152553db5"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:12.687282 2144 raft_consensus.cc:595] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:12.687602 2144 raft_consensus.cc:603] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260927 19:39:12.687925 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 310153us
W20260927 19:39:12.696581 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 249755us
I20260927 19:39:12.628712 1893 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } attrs { replace: true } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:12.747105 2683 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260927 19:39:12.757465 1988 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 LEADER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:12.757823 1893 catalog_manager.cc:5495] T aabcfbc05a724a19857afe1f8211259f P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:12.759371 1893 catalog_manager.cc:5495] T c0017d74ca734f828f8096d7d474a209 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 11 to 12, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 12 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:12.762040 1893 catalog_manager.cc:5495] T 901577b40c874966936247e62860f154 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:12.764382 1893 catalog_manager.cc:5495] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:12.756351 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:12.768024 1992 raft_consensus.cc:595] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:12.769693 1992 raft_consensus.cc:603] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:12.781613 1989 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c802eef7b2904ec984925e3999cff361"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:12.787999 1989 raft_consensus.cc:595] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:12.793979 2070 raft_consensus.cc:1261] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260927 19:39:12.794581 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 78750.2us
I20260927 19:39:12.804786 2144 raft_consensus.cc:1226] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:12.813864 2140 raft_consensus.cc:1261] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260927 19:39:12.830297 1893 catalog_manager.cc:5495] T 96289a4006ee4b44977df34d8b39798a P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: term changed from 9 to 10, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131). New cstate: current_term: 10 leader_uuid: "69f09866d0a1477585f05b2f82ff4c67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:12.835419 1992 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
mode: GRACEFUL
new_leader_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:34128
I20260927 19:39:12.835971 5478 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20260927 19:39:12.838550 1992 raft_consensus.cc:595] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Received request to transfer leadership to TS 4cc2df048f424e08803a390946004df0
I20260927 19:39:12.838955 1992 raft_consensus.cc:603] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260927 19:39:12.847358 1890 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 259) took 4047 ms. Trace:
I20260927 19:39:12.848001 1890 rpcz_store.cc:276] 0927 19:39:08.800049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:08.811799 (+ 11750us) service_pool.cc:227] Handling call
0927 19:39:09.492054 (+680255us) spinlock_profiling.cc:243] Waited 5.69 ms on lock 0x7b54002f58d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79198bcd6c 00007f79199b6c3e 00007f7917cc8fb4 00007f7917cc8f42 00007f7917cc8e71 00007f7917cc8dec 00007f7917cc80a2 00007f7911d8c94c 00007f7911d8bd86
0927 19:39:12.847297 (+3355243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":247026,"reported_tablets":35,"spinlock_wait_cycles":13083520}
I20260927 19:39:12.894794 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:39:12.896759 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:12.899209 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:12.943825 5287 consensus_queue.cc:1035] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260927 19:39:12.952679 5327 consensus_queue.cc:1035] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.042s
I20260927 19:39:12.974812 5306 consensus_queue.cc:1046] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
W20260927 19:39:13.057039 5464 log.cc:924] Time spent T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260927 19:39:13.093154 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 99892.2us
I20260927 19:39:13.093194 5120 consensus_queue.cc:1046] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260927 19:39:13.108240 5120 consensus_queue.cc:1225] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [LEADER]: Peer 3d1bc2cd9efe43399ddba396f5ddff85 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260927 19:39:13.112779 5244 raft_consensus.cc:3032] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:13.113191 5244 raft_consensus.cc:729] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:13.114020 5244 consensus_queue.cc:260] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.115437 5244 raft_consensus.cc:3037] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
W20260927 19:39:13.118026 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 126465us
I20260927 19:39:13.122720 4864 raft_consensus.cc:979] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:13.130846 2143 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260927 19:39:13.133638 4864 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:13.135074 4864 raft_consensus.cc:1067] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
W20260927 19:39:13.138665 5491 log.cc:924] Time spent T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.130s user 0.002s sys 0.000s
I20260927 19:39:13.173081 5493 raft_consensus.cc:979] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85: : Instructing follower 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:13.173663 5493 raft_consensus.cc:1067] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Signalling peer 4cc2df048f424e08803a390946004df0 to start an election
I20260927 19:39:13.178799 2070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:13.179473 2070 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:13.179581 2067 raft_consensus.cc:1261] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260927 19:39:13.179841 2070 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:13.180070 2067 pending_rounds.cc:76] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Aborting all ops after (but not including) 4
I20260927 19:39:13.180473 2067 pending_rounds.cc:97] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260927 19:39:13.180702 2067 raft_consensus.cc:2866] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260927 19:39:13.190883 2070 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.193323 2067 raft_consensus.cc:1226] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:13.193693 2066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c802eef7b2904ec984925e3999cff361"
dest_uuid: "4cc2df048f424e08803a390946004df0"
from {username='slave'} at 127.0.0.1:48556
I20260927 19:39:13.194311 2066 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:13.194679 2066 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:13.202950 2066 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.204834 2066 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:13.216208 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:13.216823 2138 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:13.221457 4864 consensus_queue.cc:1035] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260927 19:39:13.222754 4864 consensus_queue.cc:1046] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 19:39:13.224215 5497 raft_consensus.cc:3032] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Stepping down as leader of term 7
I20260927 19:39:13.224591 5497 raft_consensus.cc:729] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:13.225410 5497 consensus_queue.cc:260] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.229260 5497 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:13.250535 2138 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:13.255790 2070 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:13.260601 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:13.261785 1988 raft_consensus.cc:3032] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:13.262168 1988 raft_consensus.cc:729] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:13.263197 1988 consensus_queue.cc:260] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.264621 1988 raft_consensus.cc:3037] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:13.267565 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:13.268150 2138 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:13.268675 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:13.272262 1988 raft_consensus.cc:2454] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:13.279474 2070 raft_consensus.cc:1226] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:13.280944 1990 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:13.284379 1893 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 244) took 2393 ms. Trace:
I20260927 19:39:13.287098 1891 catalog_manager.cc:5495] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 8 to 9, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 9 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.295147 2037 leader_election.cc:302] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:13.295951 5244 raft_consensus.cc:2781] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:39:13.289425 1891 catalog_manager.cc:5495] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 8 to 9, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 9 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.298063 2138 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:13.299849 5244 raft_consensus.cc:686] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [term 5 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:13.300773 5244 consensus_queue.cc:237] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.284801 1893 rpcz_store.cc:276] 0927 19:39:10.891602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:10.894473 (+ 2871us) service_pool.cc:227] Handling call
0927 19:39:11.119181 (+224708us) spinlock_profiling.cc:243] Waited 94.7 ms on lock 0x7b54002f58d4. stack: 00007f7910a08033 00007f7910a07f30 00007f7910366534 00007f791a0728b4 00007f791a072849 00007f79138e7af0 00007f791396af2a 00007f79198bcd6c 00007f79199b6c3e 00007f7917cc8fb4 00007f7917cc8f42 00007f7917cc8e71 00007f7917cc8dec 00007f7917cc80a2 00007f7911d8c94c 00007f7911d8bd86
0927 19:39:13.284315 (+2165134us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:13.041589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:13.042364 (+ 775us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:13.042430 (+ 66us) write_op.cc:435] Acquired schema lock
0927 19:39:13.042466 (+ 36us) tablet.cc:660] Decoding operations
0927 19:39:13.043575 (+ 1109us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:13.043691 (+ 116us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:13.043731 (+ 40us) tablet.cc:683] Acquiring locks for 6 operations
0927 19:39:13.044371 (+ 640us) tablet.cc:699] Row locks acquired
0927 19:39:13.044398 (+ 27us) write_op.cc:260] PREPARE: finished
0927 19:39:13.044612 (+ 214us) write_op.cc:270] Start()
0927 19:39:13.044771 (+ 159us) write_op.cc:276] Timestamp: P: 1790537953044561 usec, L: 0
0927 19:39:13.044804 (+ 33us) op_driver.cc:331] REPLICATION: starting
0927 19:39:13.083671 (+ 38867us) log.cc:842] Serialized 2525 byte log entry
0927 19:39:13.101527 (+ 17856us) op_driver.cc:444] REPLICATION: finished
0927 19:39:13.246182 (+144655us) write_op.cc:301] APPLY: starting
0927 19:39:13.246268 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:13.246435 (+ 167us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:13.246549 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:13.247230 (+ 681us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:13.247447 (+ 217us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0927 19:39:13.247493 (+ 46us) write_op.cc:312] APPLY: finished
0927 19:39:13.247728 (+ 235us) log.cc:842] Serialized 53 byte log entry
0927 19:39:13.247885 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:13.248024 (+ 139us) write_op.cc:454] Released schema lock
0927 19:39:13.248073 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:13.248356 (+ 283us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:13.248387 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":242081,"reported_tablets":23,"spinlock_wait_cycles":217899392,"child_traces":[["op",{"apply.queue_time_us":144382,"apply.run_cpu_time_us":2560,"apply.run_wall_time_us":2560,"num_ops":6,"prepare.queue_time_us":171670,"prepare.run_cpu_time_us":4614,"prepare.run_wall_time_us":59299,"replication_time_us":53487,"thread_start_us":224051,"threads_started":2}]]}
I20260927 19:39:13.298564 1891 catalog_manager.cc:5495] T 83f97251c21d4e979bce8419c208369c P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.311908 2037 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:13.313493 5271 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader election won for term 8
I20260927 19:39:13.313973 5271 raft_consensus.cc:686] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [term 8 LEADER]: Becoming Leader. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:13.314846 5271 consensus_queue.cc:237] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:13.321974 1891 catalog_manager.cc:5495] T f19d4799cf134d51b543a2952f564adc P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:13.326977 1891 catalog_manager.cc:5495] T 794b9dbf053c445393588ff152553db5 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 8 to 9, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 9 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.335094 1891 catalog_manager.cc:5495] T c2df5ab506c34b2c974fdefdf324e734 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.336766 1891 catalog_manager.cc:5495] T ff14f359ca724a70853a0490b2f2c680 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 8 to 9, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 9 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: HEALTHY } } }
I20260927 19:39:13.349215 1891 catalog_manager.cc:5495] T 1c40cea829004940809b7bbd894de6b8 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.358285 1891 catalog_manager.cc:5495] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 6 to 7, leader changed from 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 7 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.491317 1890 catalog_manager.cc:5495] T 45a655bc810741ac9d95bf0a2cb4b759 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 7 to 8, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 8 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.531499 1890 catalog_manager.cc:5495] T da05114b545f4ec9a25e86d9c059fea3 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 5 to 6, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 6 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.543354 2688 consensus_queue.cc:237] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:13.544373 1892 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34824 (request call id 160) took 1668 ms. Trace:
I20260927 19:39:13.544867 1892 rpcz_store.cc:276] 0927 19:39:11.875495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:11.875651 (+ 156us) service_pool.cc:227] Handling call
0927 19:39:13.544307 (+1668656us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6345,"reported_tablets":15}
I20260927 19:39:13.612047 2064 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260927 19:39:13.615326 2138 raft_consensus.cc:1261] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Refusing update from remote peer 4e0335d1a85f421380029b95a91bf0c4: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260927 19:39:13.617996 5449 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 19:39:13.619057 5469 consensus_queue.cc:1035] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 19:39:13.647373 1987 consensus_queue.cc:237] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:13.651111 5469 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 LEADER]: Committing config change with OpId 9.11: config changed from index 9 to 11, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:13.663269 2140 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Committing config change with OpId 9.11: config changed from index 9 to 11, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:13.672290 1891 catalog_manager.cc:5495] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 4 to 5, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 5 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.698771 1891 catalog_manager.cc:5495] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 reported cstate change: term changed from 7 to 8, leader changed from 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) to 4cc2df048f424e08803a390946004df0 (127.0.204.130). New cstate: current_term: 8 leader_uuid: "4cc2df048f424e08803a390946004df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:13.712378 1935 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:39:13.726253 2680 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 19:39:13.728750 2070 raft_consensus.cc:1261] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260927 19:39:13.730227 5495 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 19:39:13.731379 4864 consensus_queue.cc:1035] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260927 19:39:13.766217 5509 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260927 19:39:13.794950 2143 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260927 19:39:13.858265 5244 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260927 19:39:13.889263 1891 catalog_manager.cc:5495] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 reported cstate change: config changed from index 9 to 11, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New cstate: current_term: 9 leader_uuid: "4e0335d1a85f421380029b95a91bf0c4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:39:13.926411 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 103826us
I20260927 19:39:13.972101 4864 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 LEADER]: Committing config change with OpId 9.9: config changed from index 7 to 9, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:39:13.997995 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 62143.2us
I20260927 19:39:14.005863 2138 raft_consensus.cc:1261] T c802eef7b2904ec984925e3999cff361 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260927 19:39:14.006835 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 66100.5us
I20260927 19:39:14.064258 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:14.081960 1880 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet db442f51d759423ba3bde8876ae46aa0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260927 19:39:14.145344 1990 raft_consensus.cc:1261] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260927 19:39:14.148350 1990 raft_consensus.cc:1261] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Refusing update from remote peer 4cc2df048f424e08803a390946004df0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260927 19:39:14.137213 2062 raft_consensus.cc:2932] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Committing config change with OpId 9.11: config changed from index 9 to 11, VOTER 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
W20260927 19:39:14.137652 5511 log.cc:924] Time spent T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0: Append to log took a long time: real 0.224s user 0.002s sys 0.000s
I20260927 19:39:14.174695 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 with delete_type TABLET_DATA_TOMBSTONED (TS 3d1bc2cd9efe43399ddba396f5ddff85 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:60280
I20260927 19:39:14.180692 5522 consensus_queue.cc:1035] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:14.193615 5520 consensus_queue.cc:1035] T c802eef7b2904ec984925e3999cff361 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260927 19:39:14.139164 2687 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 7 to 9, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:14.185964 1893 catalog_manager.cc:5495] T db442f51d759423ba3bde8876ae46aa0 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: config changed from index 7 to 9, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New cstate: current_term: 9 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260927 19:39:14.190393 5518 consensus_queue.cc:1035] T 3134bec8f5e84b64bcb7237586476295 P 4cc2df048f424e08803a390946004df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260927 19:39:14.342731 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 184624us
I20260927 19:39:14.373096 5534 tablet_replica.cc:330] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: stopping tablet replica
I20260927 19:39:14.374521 5534 raft_consensus.cc:2229] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Raft consensus shutting down.
I20260927 19:39:14.375351 5534 raft_consensus.cc:2258] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Raft consensus is shut down!
I20260927 19:39:14.384450 5534 ts_tablet_manager.cc:1905] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260927 19:39:14.466329 2653 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 54442.4us
W20260927 19:39:14.469457 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 110074us
W20260927 19:39:14.474484 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 62132us
W20260927 19:39:14.542253 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 471724us
I20260927 19:39:14.547576 2064 raft_consensus.cc:2932] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Committing config change with OpId 9.9: config changed from index 7 to 9, VOTER 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } } }
I20260927 19:39:14.568605 2131 tablet_service.cc:1514] Processing DeleteTablet for tablet db442f51d759423ba3bde8876ae46aa0 with delete_type TABLET_DATA_TOMBSTONED (TS 69f09866d0a1477585f05b2f82ff4c67 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40144
I20260927 19:39:14.569911 2143 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33febda9075249dc9de325af8884cce5"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:14.570508 2143 raft_consensus.cc:595] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:14.600248 5534 ts_tablet_manager.cc:1918] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 9.10
I20260927 19:39:14.619477 5534 log.cc:1196] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 3d1bc2cd9efe43399ddba396f5ddff85: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-0-root/wals/51dd0aa486b149cb9fc4a9dd54e3cdf4
I20260927 19:39:14.638769 1880 catalog_manager.cc:4841] TS 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941): tablet 51dd0aa486b149cb9fc4a9dd54e3cdf4 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
I20260927 19:39:14.640048 2064 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:39:14.640714 2064 raft_consensus.cc:595] T 7f97d36ebc9149818eb4749186e672e7 P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:14.653048 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:14.653834 2138 raft_consensus.cc:595] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:14.656581 2068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4214caa28574754b61aad89471c77b9"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:39:14.657122 2068 raft_consensus.cc:595] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:14.670701 5545 tablet_replica.cc:330] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: stopping tablet replica
I20260927 19:39:14.673564 2068 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:39:14.677624 2068 raft_consensus.cc:595] T f6ecc556dac3458bb7b5ffda19a96158 P 4cc2df048f424e08803a390946004df0 [term 8 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:14.675520 5545 raft_consensus.cc:2229] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Raft consensus shutting down.
I20260927 19:39:14.683284 818 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260927 19:39:14.685225 818 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260927 19:39:14.685941 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:14.688478 5545 raft_consensus.cc:2258] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Raft consensus is shut down!
I20260927 19:39:14.729650 5530 raft_consensus.cc:979] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0: : Instructing follower 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:39:14.730598 5530 raft_consensus.cc:1067] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Signalling peer 3d1bc2cd9efe43399ddba396f5ddff85 to start an election
I20260927 19:39:14.741605 5545 ts_tablet_manager.cc:1905] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260927 19:39:14.743216 1988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4214caa28574754b61aad89471c77b9"
dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:60296
I20260927 19:39:14.743835 1988 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 19:39:14.744174 1988 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:14.755910 1988 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:14.772226 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:14.772830 2680 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:14.773168 1988 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:14.778440 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "4cc2df048f424e08803a390946004df0"
I20260927 19:39:14.779107 2066 raft_consensus.cc:3032] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Stepping down as leader of term 4
I20260927 19:39:14.779363 2066 raft_consensus.cc:729] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4cc2df048f424e08803a390946004df0, State: Running, Role: LEADER
I20260927 19:39:14.780043 2066 consensus_queue.cc:260] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:14.781103 2066 raft_consensus.cc:3037] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:14.789922 2066 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:14.793725 2680 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:14.794993 1959 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:14.795853 5495 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election won for term 5
I20260927 19:39:14.796342 5495 raft_consensus.cc:686] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:14.797140 5495 consensus_queue.cc:237] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:14.873764 5545 ts_tablet_manager.cc:1918] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 9.8
I20260927 19:39:14.874787 5545 log.cc:1196] T db442f51d759423ba3bde8876ae46aa0 P 69f09866d0a1477585f05b2f82ff4c67: Deleting WAL directory at /tmp/dist-test-taskQPjX9t/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790537791628671-818-0/minicluster-data/ts-2-root/wals/db442f51d759423ba3bde8876ae46aa0
W20260927 19:39:16.082540 1924 debug-util.cc:398] Leaking SignalData structure 0x7b0800094be0 after lost signal to thread 821
I20260927 19:39:17.014227 5556 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.021611 5555 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.022915 5555 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.026865 5525 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.015926 5556 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.029464 5525 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.032692 5525 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.036484 5525 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.039841 5525 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.039031 5555 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.032760 5556 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.043057 5525 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.068423 5556 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.072728 5560 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.073299 5556 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.080341 5557 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.084820 5560 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.085834 5556 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.093809 5560 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.095920 5556 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.071456 5525 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.097914 5525 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.098845 5545 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40144 (request call id 666) took 2531 ms. Trace:
I20260927 19:39:17.099404 5545 rpcz_store.cc:276] 0927 19:39:14.567171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:14.568448 (+ 1277us) service_pool.cc:227] Handling call
0927 19:39:14.669984 (+101536us) ts_tablet_manager.cc:1145] Deleting tablet db442f51d759423ba3bde8876ae46aa0
0927 19:39:14.670071 (+ 87us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0927 19:39:14.804470 (+134399us) tablet_metadata.cc:641] Metadata flushed
0927 19:39:14.873219 (+ 68749us) tablet_metadata.cc:641] Metadata flushed
0927 19:39:17.098535 (+2225316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":100591,"thread_start_us":68359,"threads_started":1}
I20260927 19:39:17.100852 5525 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.096786 5556 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.081096 5557 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.110598 5559 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.111680 5559 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.113677 5102 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.114779 5102 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.121914 5561 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.123147 5561 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.129274 5560 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.137609 5562 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.138662 5562 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.139616 5559 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.134711 5561 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.130270 5560 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.125936 5525 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.129372 5531 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.178050 5556 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.191944 5561 raft_consensus.cc:482] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.192370 5560 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.198166 5525 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.212847 5555 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.213147 5525 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.213793 5525 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.213857 5555 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.214583 5525 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.215986 5567 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.218871 5559 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.219857 5559 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.199237 5556 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.200734 5102 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.217000 5525 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.222059 5556 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.230486 5525 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.230669 5531 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.231593 5570 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.199745 5560 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.222378 5555 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.232522 5556 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.216965 5567 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.242165 5556 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.243124 5556 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.245525 5562 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.246232 5562 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.247102 5562 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.245612 5560 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.245750 5567 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.248929 5567 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.249913 5560 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.251116 5557 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.251682 5563 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.252508 5567 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.252643 5563 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.260882 5556 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.262917 5559 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.263270 5563 leader_election.cc:288] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.264081 5563 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.230540 5558 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.265770 5558 raft_consensus.cc:504] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.233070 5564 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.270334 5564 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.271986 5567 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.272815 5557 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.273590 5557 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.272965 5564 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.274854 5559 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.276105 5120 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.275768 5559 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.276962 5120 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.264986 5573 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.278612 5573 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.279736 5120 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.253861 5555 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.239168 5561 raft_consensus.cc:504] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.231529 5525 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.287528 5561 leader_election.cc:288] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.254477 5533 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.288846 5533 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.289579 1889 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46386 (request call id 242) took 2527 ms. Trace:
I20260927 19:39:17.255589 5531 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.232479 5570 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.256572 5569 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.259145 5562 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.269588 5558 leader_election.cc:288] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.265060 5563 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.281972 5555 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.287562 5525 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.305482 5569 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.311857 5570 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.314014 5533 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.290060 1889 rpcz_store.cc:276] 0927 19:39:14.737476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:14.740378 (+ 2902us) service_pool.cc:227] Handling call
0927 19:39:17.265170 (+2524792us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":1408}
I20260927 19:39:17.309090 5569 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.322846 5557 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.323688 5563 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.324066 5573 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.301681 5559 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.335085 5522 raft_consensus.cc:482] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.335883 5522 raft_consensus.cc:504] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.339005 5566 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.339823 5566 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.341723 5574 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.342501 5574 raft_consensus.cc:504] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.345886 5537 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.347563 5558 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:39:17.348452 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003d5740 after lost signal to thread 1927
I20260927 19:39:17.348301 5558 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.346786 5537 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.350849 5558 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.352444 5512 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.353632 5255 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.355142 5537 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.354815 5255 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.358951 5255 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.359612 5120 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.360634 5530 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.360436 5120 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.361336 5530 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.363261 5563 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.364460 5530 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.364148 5563 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.365054 5530 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.361550 5575 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.366729 5563 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.367439 5563 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.368247 5563 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.367049 5575 raft_consensus.cc:504] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.370198 5120 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.372287 5575 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.373677 5563 leader_election.cc:288] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.375141 5522 leader_election.cc:288] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.378492 5562 raft_consensus.cc:482] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.379582 5560 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.379801 5567 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.380390 5570 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.380306 5560 raft_consensus.cc:504] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.381497 5531 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.381343 5570 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.382328 5555 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.382189 5531 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.345065 5574 leader_election.cc:288] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.382823 5560 leader_election.cc:288] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.383503 5560 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.383841 5570 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.384419 5570 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.373648 4864 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.384181 5560 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.380507 5567 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.379428 5562 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.365804 5530 raft_consensus.cc:504] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:16.696851 818 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260927 19:39:17.454954 5540 raft_consensus.cc:482] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.455979 5540 raft_consensus.cc:504] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.455116 5516 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:39:17.433409 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003e94e0 after lost signal to thread 2020
W20260927 19:39:17.462088 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003e9500 after lost signal to thread 2021
I20260927 19:39:17.386224 5566 leader_election.cc:288] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.411770 5120 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.410776 5577 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.382861 5555 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.353276 5558 raft_consensus.cc:482] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.385090 5495 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:39:17.470113 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003cb360 after lost signal to thread 2095
I20260927 19:39:17.384341 5531 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:17.515601 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003b6660 after lost signal to thread 2096
I20260927 19:39:17.378620 5525 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.458855 5557 raft_consensus.cc:482] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:39:17.524216 1924 debug-util.cc:398] Leaking SignalData structure 0x7b080030db40 after lost signal to thread 2169
I20260927 19:39:17.473079 5580 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.473089 5560 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.385182 5570 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.475018 1880 catalog_manager.cc:4841] TS 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045): tablet db442f51d759423ba3bde8876ae46aa0 (table test-workload [id=613fa53ad5754ab495032b9d9d2d4862]) successfully deleted
W20260927 19:39:17.539486 1924 debug-util.cc:398] Leaking SignalData structure 0x7b0800348480 after lost signal to thread 2170
I20260927 19:39:17.511776 4864 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.472146 5120 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.476882 5362 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.478797 5575 raft_consensus.cc:482] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
W20260927 19:39:17.544236 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003102c0 after lost signal to thread 2712
I20260927 19:39:17.483678 5542 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.474316 5555 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.481715 5348 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.480207 5522 raft_consensus.cc:482] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.524288 5530 leader_election.cc:288] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.479202 5558 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.353291 5512 raft_consensus.cc:504] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:17.561067 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08000609c0 after lost signal to thread 5493
I20260927 19:39:17.521770 5566 raft_consensus.cc:482] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.566453 5529 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.566653 5525 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.473701 5577 raft_consensus.cc:504] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.567273 5575 raft_consensus.cc:504] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.567987 5555 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.568554 5348 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.586256 4864 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.585960 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.587023 2139 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:17.588523 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.589442 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46400 (request call id 250) took 2728 ms. Trace:
I20260927 19:39:17.589692 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.589975 1891 rpcz_store.cc:276] 0927 19:39:14.786839 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:14.786965 (+ 126us) service_pool.cc:227] Handling call
0927 19:39:17.480128 (+2693163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2425488,"reported_tablets":1}
I20260927 19:39:17.539240 5580 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.593417 5519 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.543789 5557 raft_consensus.cc:504] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.595547 5565 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.596612 5565 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.597922 5583 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.598618 5583 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.599506 5480 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.600507 5480 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.601730 5585 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.602418 5585 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.602959 5559 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.604007 5559 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.601783 5557 leader_election.cc:288] T 39d534ed586444279970bb03de60c919 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.605525 5573 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.606423 5573 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.607937 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.608299 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.609493 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.609862 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.610085 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.610241 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.610576 1992 raft_consensus.cc:2454] T efcb4aec6b174f11b08d4483d9f63866 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:39:17.610697 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.593132 5542 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.609798 1996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.611147 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.611505 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.611994 1994 raft_consensus.cc:2454] T ac5251fc915042bab4ce94b1c17deb87 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 7.
I20260927 19:39:17.612087 1992 raft_consensus.cc:2454] T 29707548f67846318643f37690b25103 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:17.613567 1995 raft_consensus.cc:2454] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:17.613000 5362 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.615482 5120 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.615922 5362 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.617189 5522 raft_consensus.cc:504] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.620182 5522 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.625559 5521 raft_consensus.cc:482] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.626449 5521 raft_consensus.cc:504] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.628751 5271 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.629598 5271 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.631534 5562 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.578529 5533 raft_consensus.cc:482] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.632241 5271 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.632926 5533 raft_consensus.cc:504] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.635950 5533 leader_election.cc:288] T 3725aebc9b4c41a793e21109302696ec P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
W20260927 19:39:17.578044 1880 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 103185us
I20260927 19:39:17.560222 5495 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.543421 5581 raft_consensus.cc:482] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
W20260927 19:39:17.579512 1924 debug-util.cc:398] Leaking SignalData structure 0x7b08003a1280 after lost signal to thread 5539
I20260927 19:39:17.610837 1987 raft_consensus.cc:2454] T 3d15dfddc2cb4edfaef7625c7eba492c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:39:17.549237 5569 raft_consensus.cc:482] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.544696 5567 leader_election.cc:288] T 228a1cf257434cbe9e5616dbd94e1864 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.556118 5255 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:17.594304 5519 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.609071 5573 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.610697 1990 raft_consensus.cc:2454] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 8.
I20260927 19:39:17.518880 5516 raft_consensus.cc:504] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.660483 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.577668 5564 raft_consensus.cc:482] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:17.661240 2139 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:17.659041 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.675377 5582 raft_consensus.cc:482] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.677522 5564 raft_consensus.cc:504] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.688908 5348 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.689182 5540 leader_election.cc:288] T 1c40cea829004940809b7bbd894de6b8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.686918 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.689617 5577 leader_election.cc:288] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.689828 5542 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.690459 5587 raft_consensus.cc:482] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.688911 5580 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.690855 5588 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.689651 5565 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.660277 5566 raft_consensus.cc:504] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.693602 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.664930 5522 raft_consensus.cc:482] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.694764 5566 leader_election.cc:288] T 4667efc6973d414fbff5011f6ceaa9b1 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.694707 5522 raft_consensus.cc:504] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.702135 5481 raft_consensus.cc:482] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.703938 5522 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.695907 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.704879 2138 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:17.706534 5519 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.696939 5516 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.697695 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.698645 5537 raft_consensus.cc:482] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.699539 5583 leader_election.cc:288] T 81be6adca85345a3b402f5d474ec3e5a P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.701134 5589 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.701215 5469 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.702569 5495 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.695240 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.695104 2139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.658598 5569 raft_consensus.cc:504] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.705076 5584 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.680061 5255 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.731792 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.715087 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.714712 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.676328 5582 raft_consensus.cc:504] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.730608 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.718223 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.795548 2137 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:17.795140 2064 raft_consensus.cc:2454] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
I20260927 19:39:17.683550 5529 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.642127 5581 raft_consensus.cc:504] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.703999 5481 raft_consensus.cc:504] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.691787 5588 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.716900 5563 raft_consensus.cc:482] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.717295 5244 raft_consensus.cc:482] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.793524 5593 raft_consensus.cc:482] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.723157 5525 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.717519 5120 raft_consensus.cc:482] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.817284 5593 raft_consensus.cc:504] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.817955 5120 raft_consensus.cc:504] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.819731 5593 leader_election.cc:288] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.819828 5120 leader_election.cc:288] T 96289a4006ee4b44977df34d8b39798a P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.724268 5570 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.735173 5559 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.792662 5591 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.817147 5244 raft_consensus.cc:504] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.821835 5591 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.822916 5481 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.823230 5244 leader_election.cc:288] T a2ed94cb8c6c4a0badc3ff336e7b2968 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.823585 5481 raft_consensus.cc:482] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.825852 5591 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.716897 5575 leader_election.cc:288] T c802eef7b2904ec984925e3999cff361 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.724720 5569 leader_election.cc:288] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.826974 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.811175 5589 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.712124 5521 leader_election.cc:288] T f30da008e35141e99c8237c84d8e66b8 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.716655 5527 raft_consensus.cc:482] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.830278 5589 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.717692 5574 raft_consensus.cc:482] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.830528 5255 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.794818 5480 leader_election.cc:288] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.831089 5574 raft_consensus.cc:504] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.832105 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.832798 2064 raft_consensus.cc:2454] T ba36610baf2046bc9b0bd15df9129f16 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:17.833539 5574 leader_election.cc:288] T d9ad83fb0df9449f9081439026d2964f P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.719580 5577 raft_consensus.cc:482] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.833745 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.834511 2064 raft_consensus.cc:2454] T d0f797e07d634f6abe29556a11863581 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:17.834450 5577 raft_consensus.cc:504] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.835531 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.836899 5577 leader_election.cc:288] T acb3a2a0b9b047249cd5a7535a946fe9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.715984 5558 leader_election.cc:288] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.767390 5585 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.825752 5481 raft_consensus.cc:504] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.830072 5527 raft_consensus.cc:504] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.791698 5592 raft_consensus.cc:482] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:17.836750 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.837056 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.809249 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.846691 2067 raft_consensus.cc:2454] T 4fe684e9563d491b9d4b3b99204450c8 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:17.718025 5512 leader_election.cc:288] T 3c4fa5fe3da244bb9e845557555810b8 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.846889 2065 raft_consensus.cc:2454] T 1b0e603e501b4a8e9e2fe63d3784d744 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:17.691344 5587 raft_consensus.cc:504] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.767064 5537 raft_consensus.cc:504] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.846127 2145 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:17.720036 5469 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.848618 5564 leader_election.cc:288] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.863240 5592 raft_consensus.cc:504] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.864238 5527 leader_election.cc:288] T 32bc1524f46f4197b6ac8d29ea44ab77 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.866285 5581 leader_election.cc:288] T 876ffc23594245239e2688f5992774cf P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:17.807020 5584 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.866719 5537 leader_election.cc:288] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.868403 5582 leader_election.cc:288] T 901577b40c874966936247e62860f154 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.868685 5584 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.863256 5594 raft_consensus.cc:482] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.871017 5469 leader_election.cc:288] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.870850 5594 raft_consensus.cc:504] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.874229 5529 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.875069 5588 leader_election.cc:288] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.875248 5481 leader_election.cc:288] T 63522eab31e04c8191338bb22acb0297 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.843118 1891 catalog_manager.cc:5495] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 reported cstate change: term changed from 4 to 5, leader changed from 4cc2df048f424e08803a390946004df0 (127.0.204.130) to 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129). New cstate: current_term: 5 leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260927 19:39:17.877748 5587 leader_election.cc:288] T 069da74585d54798b7b8d625c46ae785 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.881508 5592 leader_election.cc:288] T 6ec895e575154c488c4fb6666ae75ce3 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.882756 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.883507 1989 raft_consensus.cc:2454] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 7.
I20260927 19:39:17.852939 5591 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 19:39:17.891316 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.896821 2069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.897542 2069 raft_consensus.cc:2454] T 85f45e6282f545f19d655ea989e9e123 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:17.899725 2140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.900882 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.900964 2144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5251fc915042bab4ce94b1c17deb87" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.901181 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.901998 2142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.903424 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.906263 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d15dfddc2cb4edfaef7625c7eba492c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.854699 5563 raft_consensus.cc:504] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.904551 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.910357 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.913801 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.914997 1994 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
I20260927 19:39:17.916461 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.915659 1962 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:17.918118 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.918185 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b0e603e501b4a8e9e2fe63d3784d744" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.919133 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.919332 5563 leader_election.cc:288] T 33febda9075249dc9de325af8884cce5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.919817 2066 raft_consensus.cc:2454] T 8814088a5337420ab29ac2b4b4e0f314 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:17.923928 2068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.925423 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
W20260927 19:39:17.926384 1962 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 82131.5us
I20260927 19:39:17.928668 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.927268 5575 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:17.917428 5592 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:17.930492 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.931006 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.931778 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.930565 5575 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.932869 5575 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:17.941498 5598 raft_consensus.cc:482] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.942952 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cc0a374f434fc193a1c4fc0edcf8c5" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.943696 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.944383 2136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.945089 2136 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 5.
I20260927 19:39:17.945181 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.945577 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.946085 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.946300 2141 raft_consensus.cc:2454] T 3134bec8f5e84b64bcb7237586476295 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 8.
I20260927 19:39:17.944422 2145 raft_consensus.cc:2454] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:17.941721 5594 leader_election.cc:288] T f19d4799cf134d51b543a2952f564adc P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.948539 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c802eef7b2904ec984925e3999cff361" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.949793 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.950197 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "794b9dbf053c445393588ff152553db5" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.931001 5592 raft_consensus.cc:504] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.949841 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.953438 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.954123 2137 raft_consensus.cc:2454] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 9.
I20260927 19:39:17.913592 5591 raft_consensus.cc:504] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.956508 5591 leader_election.cc:288] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.959190 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.959844 2062 raft_consensus.cc:2454] T 9e0ad2bfdf76425782a9548c96da6ca7 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:17.959923 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.960675 2065 raft_consensus.cc:2454] T 45a655bc810741ac9d95bf0a2cb4b759 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 8.
I20260927 19:39:17.960758 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.961513 2066 raft_consensus.cc:2454] T 81be6adca85345a3b402f5d474ec3e5a P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:17.958839 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.962131 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.962210 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.962530 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.964394 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.964499 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.965049 2687 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:17.965334 2065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:17.956203 5575 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:17.966055 2065 raft_consensus.cc:2454] T 876ffc23594245239e2688f5992774cf P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 6.
I20260927 19:39:17.966667 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.967060 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.967679 1990 raft_consensus.cc:2454] T 2034953084704d748cf5bd91b19f1a46 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 6.
I20260927 19:39:17.942361 5598 raft_consensus.cc:504] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:17.917021 5597 raft_consensus.cc:482] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:17.972873 5575 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:17.974843 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.975589 1994 raft_consensus.cc:2454] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 10.
I20260927 19:39:17.977526 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.980656 5592 leader_election.cc:288] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941)
I20260927 19:39:17.982511 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2034953084704d748cf5bd91b19f1a46" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.983278 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d534ed586444279970bb03de60c919" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.986537 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.986856 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.988212 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.989814 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.990056 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.973213 5598 leader_election.cc:288] T f6ecc556dac3458bb7b5ffda19a96158 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:17.990794 2687 raft_consensus.cc:2454] T 9e017d73273c4a83bcfdb4c95caa32df P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 4.
I20260927 19:39:17.989212 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.991943 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:17.990546 2688 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4e0335d1a85f421380029b95a91bf0c4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 9.
I20260927 19:39:17.994858 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97d36ebc9149818eb4749186e672e7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.995673 1990 raft_consensus.cc:2454] T 7f97d36ebc9149818eb4749186e672e7 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 9.
I20260927 19:39:17.996933 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.997810 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:17.998507 2141 raft_consensus.cc:2454] T c2df5ab506c34b2c974fdefdf324e734 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 7.
I20260927 19:39:18.008013 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:17.972802 5597 raft_consensus.cc:504] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.011816 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.012733 5597 leader_election.cc:288] T b2f6360a6b784953bc89cd72712a5e7d P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:18.015450 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.018831 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.022279 1991 raft_consensus.cc:1226] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 7. Current term is 8. Ops: []
I20260927 19:39:18.023540 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.022324 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.024468 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.027266 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.028558 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.029523 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.032095 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.032356 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.032697 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.033428 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.033697 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.033998 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.034651 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.034943 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.035290 1994 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 5.
I20260927 19:39:18.037230 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c40cea829004940809b7bbd894de6b8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.037760 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.040177 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.037178 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.048039 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.047938 1962 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.049263 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.050287 1962 leader_election.cc:302] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.050946 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.052245 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:18.057807 5570 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:18.058493 5570 raft_consensus.cc:482] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.059047 5570 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:18.060585 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.061275 2679 raft_consensus.cc:2454] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 4.
I20260927 19:39:18.061674 5575 raft_consensus.cc:2781] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:18.062235 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.061748 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d811e58acb4b4a3aa93ab155fd5ab1c2" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.062463 5575 raft_consensus.cc:482] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.062482 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.063040 5575 raft_consensus.cc:3037] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:18.063309 2687 raft_consensus.cc:2454] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 12.
I20260927 19:39:18.064880 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.065094 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e017d73273c4a83bcfdb4c95caa32df" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.066044 1962 leader_election.cc:302] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.066898 5525 raft_consensus.cc:2781] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.067075 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0017d74ca734f828f8096d7d474a209" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.067473 5525 raft_consensus.cc:482] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.067831 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.068024 5525 raft_consensus.cc:3037] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:18.068552 1962 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.069291 5495 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.069327 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.069832 5495 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.070261 5495 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:18.071455 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51dd0aa486b149cb9fc4a9dd54e3cdf4" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.073673 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63522eab31e04c8191338bb22acb0297" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.074452 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8814088a5337420ab29ac2b4b4e0f314" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.077924 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.078301 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.080539 1962 leader_election.cc:302] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.081410 5586 raft_consensus.cc:2781] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:18.081904 5586 raft_consensus.cc:482] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.082321 5586 raft_consensus.cc:3037] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:18.092561 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.113885 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.167404 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4667efc6973d414fbff5011f6ceaa9b1" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.118992 1987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.120066 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.116379 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.166782 1993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.166622 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.173415 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81be6adca85345a3b402f5d474ec3e5a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.175859 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.179596 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.179530 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.179549 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96289a4006ee4b44977df34d8b39798a" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.179694 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f30da008e35141e99c8237c84d8e66b8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.178156 1995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.182646 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.185606 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45a655bc810741ac9d95bf0a2cb4b759" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.187692 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.189177 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.114763 5570 raft_consensus.cc:504] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.158767 5575 raft_consensus.cc:504] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.190092 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ad83fb0df9449f9081439026d2964f" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.190672 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.193681 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed94cb8c6c4a0badc3ff336e7b2968" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.141052 5495 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.196408 5570 leader_election.cc:288] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:18.197239 5495 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:18.198462 5575 leader_election.cc:288] T 3134bec8f5e84b64bcb7237586476295 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 9 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:18.200256 1989 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:18.200903 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f61659c573e142aa9c3fbb673f3e15f0" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:18.201620 2143 raft_consensus.cc:3037] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:18.198462 2111 leader_election.cc:302] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.205483 2067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:18.207945 2145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:18.174865 5525 raft_consensus.cc:504] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.207970 2064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:18.212828 2064 raft_consensus.cc:2454] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 9.
I20260927 19:39:18.212921 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.210047 2137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3134bec8f5e84b64bcb7237586476295" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:18.211771 2066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:18.211848 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32bc1524f46f4197b6ac8d29ea44ab77" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.216295 2688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba36610baf2046bc9b0bd15df9129f16" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.217872 2062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4cc2df048f424e08803a390946004df0" is_pre_election: true
I20260927 19:39:18.218575 5525 leader_election.cc:288] T 4f0cc9e9ed38443aa7b5b0a629befdcd P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 10 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:18.218729 2062 raft_consensus.cc:2454] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e0335d1a85f421380029b95a91bf0c4 in term 6.
I20260927 19:39:18.219586 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb3a2a0b9b047249cd5a7535a946fe9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.220096 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.221217 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.221802 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.214658 5480 raft_consensus.cc:2781] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:18.223223 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f45e6282f545f19d655ea989e9e123" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.216388 2111 leader_election.cc:302] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.221907 2686 raft_consensus.cc:2454] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc2df048f424e08803a390946004df0 in term 5.
I20260927 19:39:18.222679 5480 raft_consensus.cc:482] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.228376 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.228636 5480 raft_consensus.cc:3037] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:18.229863 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "901577b40c874966936247e62860f154" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.231355 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcc7a171b3e443d8f28fd70f8742470" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.228977 5559 raft_consensus.cc:2781] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.232617 5559 raft_consensus.cc:482] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.232967 5559 raft_consensus.cc:3037] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:18.234653 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.237406 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069da74585d54798b7b8d625c46ae785" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.237709 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0cc9e9ed38443aa7b5b0a629befdcd" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
W20260927 19:39:18.241544 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 75758us
I20260927 19:39:18.247108 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.247687 2680 raft_consensus.cc:2454] T 303e2824ebc54b4c9bce740f15e0105d P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69f09866d0a1477585f05b2f82ff4c67 in term 4.
I20260927 19:39:18.252178 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fe684e9563d491b9d4b3b99204450c8" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.246037 2682 raft_consensus.cc:1261] T a4214caa28574754b61aad89471c77b9 P 4e0335d1a85f421380029b95a91bf0c4 [term 5 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260927 19:39:18.252985 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.254812 2138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33febda9075249dc9de325af8884cce5" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.270084 2143 raft_consensus.cc:2454] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in term 6.
I20260927 19:39:18.270287 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ad2bfdf76425782a9548c96da6ca7" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.270823 2141 raft_consensus.cc:1226] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Rejecting Update request from peer 4cc2df048f424e08803a390946004df0 for earlier term 5. Current term is 6. Ops: []
I20260927 19:39:18.278553 1962 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.280452 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29707548f67846318643f37690b25103" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.281483 5525 raft_consensus.cc:2781] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader election won for term 6
I20260927 19:39:18.284041 5525 raft_consensus.cc:686] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Leader. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:18.285054 5525 consensus_queue.cc:237] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.292114 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.292466 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.293990 2687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228a1cf257434cbe9e5616dbd94e1864" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.294709 2684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876ffc23594245239e2688f5992774cf" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.294948 2654 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:18.295897 2681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac63cb1bcdd141bb9de9fcb5d0f102af" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.296725 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6886530f9f741c2acaff96949f65f19" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.297035 5597 raft_consensus.cc:2781] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260927 19:39:18.297801 5597 raft_consensus.cc:482] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.298204 5597 raft_consensus.cc:3037] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Advancing to term 11
I20260927 19:39:18.291019 2680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acf4c2d78fb41ad974812ebf41d0478" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.250444 5586 raft_consensus.cc:504] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.303133 2037 leader_election.cc:302] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:18.304049 5530 raft_consensus.cc:2781] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:18.304665 5530 raft_consensus.cc:482] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.305024 5530 raft_consensus.cc:3037] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:18.305055 5586 leader_election.cc:288] T c2df5ab506c34b2c974fdefdf324e734 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 8 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
W20260927 19:39:18.306183 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 75087.7us
I20260927 19:39:18.307164 2034 leader_election.cc:302] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.307904 5563 raft_consensus.cc:2781] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:18.308293 5563 raft_consensus.cc:482] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:18.308629 5563 raft_consensus.cc:3037] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:18.304010 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fa5fe3da244bb9e845557555810b8" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.310868 5480 raft_consensus.cc:504] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.306188 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
W20260927 19:39:18.313809 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 83742.5us
I20260927 19:39:18.314793 2037 leader_election.cc:302] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:18.315515 5587 raft_consensus.cc:2781] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:18.315876 5587 raft_consensus.cc:482] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.316084 1959 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.316403 5587 raft_consensus.cc:3037] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:18.319751 5575 raft_consensus.cc:2781] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:18.320190 5575 raft_consensus.cc:482] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.333910 2037 leader_election.cc:302] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:18.320091 1959 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4; no voters: 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.315799 2034 leader_election.cc:302] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.337409 2141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2df5ab506c34b2c974fdefdf324e734" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:18.339684 2034 leader_election.cc:302] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.344480 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ec895e575154c488c4fb6666ae75ce3" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4" is_pre_election: true
I20260927 19:39:18.320710 5575 raft_consensus.cc:3037] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:18.347342 2037 leader_election.cc:302] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:18.320811 5480 leader_election.cc:288] T 83f97251c21d4e979bce8419c208369c P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:18.345039 5582 raft_consensus.cc:2781] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.344508 5567 raft_consensus.cc:2781] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260927 19:39:18.338254 5563 raft_consensus.cc:504] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.349090 5597 raft_consensus.cc:504] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
W20260927 19:39:18.364318 2111 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 77866.3us
I20260927 19:39:18.344336 5590 raft_consensus.cc:2781] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:18.346395 5271 raft_consensus.cc:2781] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.362782 5530 raft_consensus.cc:504] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.364019 2110 leader_election.cc:302] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.375144 2110 leader_election.cc:302] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.361073 5582 raft_consensus.cc:482] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.368950 5271 raft_consensus.cc:482] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:18.361330 5567 raft_consensus.cc:482] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.377638 5480 raft_consensus.cc:2781] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.383975 5271 raft_consensus.cc:3037] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:18.384037 5563 leader_election.cc:288] T da05114b545f4ec9a25e86d9c059fea3 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:18.386816 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4214caa28574754b61aad89471c77b9" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.380327 2110 leader_election.cc:302] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.388522 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecc556dac3458bb7b5ffda19a96158" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.390190 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.391551 1994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2f6360a6b784953bc89cd72712a5e7d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" is_pre_election: true
I20260927 19:39:18.369606 5577 raft_consensus.cc:2781] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:18.398389 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabcfbc05a724a19857afe1f8211259f" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:18.369808 5590 raft_consensus.cc:482] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
W20260927 19:39:18.403674 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 60518.8us
I20260927 19:39:18.397048 2111 leader_election.cc:302] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.402267 5582 raft_consensus.cc:3037] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 9 FOLLOWER]: Advancing to term 10
W20260927 19:39:18.433921 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 53712.5us
I20260927 19:39:18.401054 5567 raft_consensus.cc:3037] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 12 FOLLOWER]: Advancing to term 13
I20260927 19:39:18.397786 5529 raft_consensus.cc:2781] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:18.435684 5564 raft_consensus.cc:2781] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:18.442181 5564 raft_consensus.cc:482] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:18.441921 5559 raft_consensus.cc:504] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.442579 5564 raft_consensus.cc:3037] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:18.443945 5559 leader_election.cc:288] T ff14f359ca724a70853a0490b2f2c680 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:18.401153 5480 raft_consensus.cc:482] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.445593 5480 raft_consensus.cc:3037] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:18.437719 2110 leader_election.cc:302] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.431751 5597 leader_election.cc:288] T aabcfbc05a724a19857afe1f8211259f P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 11 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:18.441735 5530 leader_election.cc:288] T efcb4aec6b174f11b08d4483d9f63866 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:18.426959 5577 raft_consensus.cc:482] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.431200 5590 raft_consensus.cc:3037] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
W20260927 19:39:18.463692 2037 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.129:39941, user_credentials={real_user=slave}} blocked reactor thread for 116407us
I20260927 19:39:18.435745 5601 raft_consensus.cc:2781] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260927 19:39:18.453724 2110 leader_election.cc:302] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0
I20260927 19:39:18.450872 5529 raft_consensus.cc:482] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.471084 5601 raft_consensus.cc:482] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4cc2df048f424e08803a390946004df0)
I20260927 19:39:18.455484 5577 raft_consensus.cc:3037] T 56cc0a374f434fc193a1c4fc0edcf8c5 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Advancing to term 6
I20260927 19:39:18.485893 5529 raft_consensus.cc:3037] T ba36610baf2046bc9b0bd15df9129f16 P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:18.465063 5559 raft_consensus.cc:2781] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:18.493464 5559 raft_consensus.cc:482] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.493958 5559 raft_consensus.cc:3037] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Advancing to term 7
I20260927 19:39:18.496752 5604 raft_consensus.cc:2726] T 794b9dbf053c445393588ff152553db5 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:18.491199 5601 raft_consensus.cc:3037] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Advancing to term 10
I20260927 19:39:18.502329 5559 raft_consensus.cc:504] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.509057 2111 leader_election.cc:302] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67; no voters:
I20260927 19:39:18.508656 5575 raft_consensus.cc:504] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.512279 2037 leader_election.cc:302] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:18.524312 2037 leader_election.cc:302] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:18.392680 5587 raft_consensus.cc:504] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.509303 5559 leader_election.cc:288] T 4fe684e9563d491b9d4b3b99204450c8 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:18.534116 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff14f359ca724a70853a0490b2f2c680" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:18.537928 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f97251c21d4e979bce8419c208369c" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
W20260927 19:39:18.542449 2108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 132239us
I20260927 19:39:18.543757 2108 leader_election.cc:302] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:18.537077 5564 raft_consensus.cc:504] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.545686 2143 raft_consensus.cc:1226] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Rejecting Update request from peer 3d1bc2cd9efe43399ddba396f5ddff85 for earlier term 6. Current term is 7. Ops: []
I20260927 19:39:18.546984 5575 leader_election.cc:288] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Requested vote from peers 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:18.548447 5564 leader_election.cc:288] T d0f797e07d634f6abe29556a11863581 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:18.487919 5570 consensus_queue.cc:1035] T a4214caa28574754b61aad89471c77b9 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260927 19:39:18.530711 5563 raft_consensus.cc:2781] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:18.551203 5563 raft_consensus.cc:482] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.551623 5563 raft_consensus.cc:3037] T d811e58acb4b4a3aa93ab155fd5ab1c2 P 4cc2df048f424e08803a390946004df0 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:18.551847 5587 leader_election.cc:288] T ac5251fc915042bab4ce94b1c17deb87 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 8 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 69f09866d0a1477585f05b2f82ff4c67 (127.0.204.131:45045)
I20260927 19:39:18.537801 5480 raft_consensus.cc:504] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.554330 5480 leader_election.cc:288] T 51dd0aa486b149cb9fc4a9dd54e3cdf4 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:18.550325 5606 raft_consensus.cc:2781] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:18.555946 5606 raft_consensus.cc:482] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.556217 5480 raft_consensus.cc:2726] T 63522eab31e04c8191338bb22acb0297 P 69f09866d0a1477585f05b2f82ff4c67 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260927 19:39:18.556334 5606 raft_consensus.cc:3037] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:18.558531 1989 raft_consensus.cc:2379] T 83f97251c21d4e979bce8419c208369c P 3d1bc2cd9efe43399ddba396f5ddff85 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 69f09866d0a1477585f05b2f82ff4c67 in current term 8: Already voted for candidate 3d1bc2cd9efe43399ddba396f5ddff85 in this term.
I20260927 19:39:18.558807 5271 raft_consensus.cc:504] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.536799 5559 raft_consensus.cc:2741] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260927 19:39:18.562099 2654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.131:45045, user_credentials={real_user=slave}} blocked reactor thread for 124623us
W20260927 19:39:18.563043 1959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 134815us
I20260927 19:39:18.563947 1959 leader_election.cc:302] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:18.479959 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 125433us
I20260927 19:39:18.567476 2037 leader_election.cc:302] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4cc2df048f424e08803a390946004df0; no voters:
I20260927 19:39:18.591768 2654 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
W20260927 19:39:18.593247 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 102921us
I20260927 19:39:18.566586 5495 raft_consensus.cc:2726] T aabcfbc05a724a19857afe1f8211259f P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 19:39:18.597756 2110 leader_election.cc:302] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.600486 5616 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 4e0335d1a85f421380029b95a91bf0c4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 19:39:18.600308 2654 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.601528 5616 raft_consensus.cc:2726] T c6886530f9f741c2acaff96949f65f19 P 4e0335d1a85f421380029b95a91bf0c4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:18.594198 1959 leader_election.cc:302] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.602557 2654 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.603287 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
I20260927 19:39:18.603370 2034 leader_election.cc:302] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 4e0335d1a85f421380029b95a91bf0c4; no voters:
I20260927 19:39:18.606976 2654 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.590837 5607 raft_consensus.cc:2781] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260927 19:39:18.608510 1959 leader_election.cc:302] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.609756 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19d4799cf134d51b543a2952f564adc" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.610845 2654 leader_election.cc:302] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.615983 2108 leader_election.cc:302] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:18.607637 5495 raft_consensus.cc:2726] T 3acf4c2d78fb41ad974812ebf41d0478 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260927 19:39:18.613826 5605 raft_consensus.cc:2726] T 6ec895e575154c488c4fb6666ae75ce3 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260927 19:39:18.608984 5592 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 4e0335d1a85f421380029b95a91bf0c4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:18.620838 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f797e07d634f6abe29556a11863581" candidate_uuid: "69f09866d0a1477585f05b2f82ff4c67" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
I20260927 19:39:18.622064 5559 raft_consensus.cc:2781] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260927 19:39:18.622505 5559 raft_consensus.cc:482] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:18.622901 5559 raft_consensus.cc:3037] T 1b0e603e501b4a8e9e2fe63d3784d744 P 69f09866d0a1477585f05b2f82ff4c67 [term 7 FOLLOWER]: Advancing to term 8
I20260927 19:39:18.623023 1992 raft_consensus.cc:1226] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 4. Current term is 5. Ops: []
I20260927 19:39:18.622303 2685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0505ba7a2573409cbd518d77ccbcd201" candidate_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:18.619544 5271 leader_election.cc:288] T db442f51d759423ba3bde8876ae46aa0 P 4cc2df048f424e08803a390946004df0 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4e0335d1a85f421380029b95a91bf0c4 (127.0.204.132:45629)
I20260927 19:39:18.641917 2143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "303e2824ebc54b4c9bce740f15e0105d" candidate_uuid: "4e0335d1a85f421380029b95a91bf0c4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "69f09866d0a1477585f05b2f82ff4c67" is_pre_election: true
I20260927 19:39:18.614387 5607 raft_consensus.cc:482] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 69f09866d0a1477585f05b2f82ff4c67)
I20260927 19:39:18.628892 5594 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 4e0335d1a85f421380029b95a91bf0c4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260927 19:39:18.631501 5527 raft_consensus.cc:2781] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260927 19:39:18.650512 2108 leader_election.cc:302] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.631675 5591 raft_consensus.cc:2726] T c0017d74ca734f828f8096d7d474a209 P 4e0335d1a85f421380029b95a91bf0c4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260927 19:39:18.608255 5525 consensus_queue.cc:1046] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: INVALID_TERM, Last received: 6.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260927 19:39:18.656158 1959 leader_election.cc:302] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d1bc2cd9efe43399ddba396f5ddff85; no voters: 4e0335d1a85f421380029b95a91bf0c4, 69f09866d0a1477585f05b2f82ff4c67
I20260927 19:39:18.659214 2108 leader_election.cc:302] T ac63cb1bcdd141bb9de9fcb5d0f102af P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
W20260927 19:39:18.658104 2034 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.132:45629, user_credentials={real_user=slave}} blocked reactor thread for 54803us
I20260927 19:39:18.669450 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db442f51d759423ba3bde8876ae46aa0" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:18.669744 2679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efcb4aec6b174f11b08d4483d9f63866" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:18.652484 5480 raft_consensus.cc:2726] T 0fcc7a171b3e443d8f28fd70f8742470 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260927 19:39:18.673485 2108 leader_election.cc:302] T c6886530f9f741c2acaff96949f65f19 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 69f09866d0a1477585f05b2f82ff4c67; no voters: 3d1bc2cd9efe43399ddba396f5ddff85, 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:18.653460 5527 raft_consensus.cc:482] T a4214caa28574754b61aad89471c77b9 P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 19:39:18.651598 5607 raft_consensus.cc:3037] T 2034953084704d748cf5bd91b19f1a46 P 4cc2df048f424e08803a390946004df0 [term 6 FOLLOWER]: Advancing to term 7
W20260927 19:39:18.681493 2110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.204.130:45515, user_credentials={real_user=slave}} blocked reactor thread for 83829.7us
I20260927 19:39:18.685989 2686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da05114b545f4ec9a25e86d9c059fea3" candidate_uuid: "4cc2df048f424e08803a390946004df0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4e0335d1a85f421380029b95a91bf0c4"
I20260927 19:39:18.658524 5608 raft_consensus.cc:2781] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260927 19:39:18.658648 5601 raft_consensus.cc:504] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.705791 5606 raft_consensus.cc:504] T 3d15dfddc2cb4edfaef7625c7eba492c P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } }
I20260927 19:39:18.711886 5601 leader_election.cc:288] T 7f97d36ebc9149818eb4749186e672e7 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 10 election: Requested vote from peers 3d1bc2cd9efe43399ddba396f5ddff85 (127.0.204.129:39941), 4cc2df048f424e08803a390946004df0 (127.0.204.130:45515)
I20260927 19:39:18.642027 5582 raft_consensus.cc:504] T 29707548f67846318643f37690b25103 P 4cc2df048f424e08803a390946004df0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.712551 2138 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33febda9075249dc9de325af8884cce5"
dest_uuid: "69f09866d0a1477585f05b2f82ff4c67"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:56794
I20260927 19:39:18.693217 5495 raft_consensus.cc:3032] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Stepping down as leader of term 6
I20260927 19:39:18.713320 2138 raft_consensus.cc:595] T 33febda9075249dc9de325af8884cce5 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.713472 5495 raft_consensus.cc:729] T d0f797e07d634f6abe29556a11863581 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d1bc2cd9efe43399ddba396f5ddff85, State: Running, Role: LEADER
I20260927 19:39:18.695971 2143 raft_consensus.cc:1261] T f61659c573e142aa9c3fbb673f3e15f0 P 69f09866d0a1477585f05b2f82ff4c67 [term 6 FOLLOWER]: Refusing update from remote peer 3d1bc2cd9efe43399ddba396f5ddff85: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260927 19:39:18.697432 1891 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46382 (request call id 267) took 1036 ms. Trace:
I20260927 19:39:18.697171 5590 raft_consensus.cc:504] T 9e017d73273c4a83bcfdb4c95caa32df P 4cc2df048f424e08803a390946004df0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "4cc2df048f424e08803a390946004df0" member_type: VOTER last_known_addr { host: "127.0.204.130" port: 45515 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.700289 2110 leader_election.cc:302] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4cc2df048f424e08803a390946004df0, 69f09866d0a1477585f05b2f82ff4c67; no voters: 4e0335d1a85f421380029b95a91bf0c4
I20260927 19:39:18.701107 5567 raft_consensus.cc:504] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129" port: 39941 } } peers { permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 } } peers { permanent_uuid: "4e0335d1a85f421380029b95a91bf0c4" member_type: VOTER last_known_addr { host: "127.0.204.132" port: 45629 } attrs { promote: false } }
I20260927 19:39:18.720659 5480 raft_consensus.cc:2781] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260927 19:39:18.721275 5480 raft_consensus.cc:482] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4e0335d1a85f421380029b95a91bf0c4)
I20260927 19:39:18.721966 5480 raft_consensus.cc:3037] T 85f45e6282f545f19d655ea989e9e123 P 69f09866d0a1477585f05b2f82ff4c67 [term 8 FOLLOWER]: Advancing to term 9
I20260927 19:39:18.722106 5575 consensus_queue.cc:1035] T f61659c573e142aa9c3fbb673f3e15f0 P 3d1bc2cd9efe43399ddba396f5ddff85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f09866d0a1477585f05b2f82ff4c67" member_type: VOTER last_known_addr { host: "127.0.204.131" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260927 19:39:18.723282 2062 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3725aebc9b4c41a793e21109302696ec"
dest_uuid: "4cc2df048f424e08803a390946004df0"
mode: GRACEFUL
new_leader_uuid: "3d1bc2cd9efe43399ddba396f5ddff85"
from {username='slave'} at 127.0.0.1:48440
I20260927 19:39:18.723878 2062 raft_consensus.cc:595] T 3725aebc9b4c41a793e21109302696ec P 4cc2df048f424e08803a390946004df0 [term 9 LEADER]: Received request to transfer leadership to TS 3d1bc2cd9efe43399ddba396f5ddff85
I20260927 19:39:18.715936 1891 rpcz_store.cc:276] 0927 19:39:17.709162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0927 19:39:17.714244 (+ 5082us) service_pool.cc:227] Handling call
0927 19:39:18.697315 (+983071us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0927 19:39:17.968512 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0927 19:39:17.969270 (+ 758us) write_op.cc:432] Acquiring schema lock in shared mode
0927 19:39:17.969418 (+ 148us) write_op.cc:435] Acquired schema lock
0927 19:39:17.969456 (+ 38us) tablet.cc:660] Decoding operations
0927 19:39:17.969957 (+ 501us) write_op.cc:620] Acquiring the partition lock for write op
0927 19:39:17.970079 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0927 19:39:17.970133 (+ 54us) tablet.cc:683] Acquiring locks for 1 operations
0927 19:39:17.970569 (+ 436us) tablet.cc:699] Row locks acquired
0927 19:39:17.970607 (+ 38us) write_op.cc:260] PREPARE: finished
0927 19:39:17.970788 (+ 181us) write_op.cc:270] Start()
0927 19:39:17.970964 (+ 176us) write_op.cc:276] Timestamp: P: 1790537957970726 usec, L: 0
0927 19:39:17.971012 (+ 48us) op_driver.cc:331] REPLICATION: starting
0927 19:39:17.971825 (+ 813us) log.cc:842] Serialized 536 byte log entry
0927 19:39:18.541085 (+569260us) op_driver.cc:444] REPLICATION: finished
0927 19:39:18.693754 (+152669us) write_op.cc:301] APPLY: starting
0927 19:39:18.693860 (+ 106us) tablet.cc:1365] starting BulkCheckPresence
0927 19:39:18.694041 (+ 181us) tablet.cc:1368] finished BulkCheckPresence
0927 19:39:18.694162 (+ 121us) tablet.cc:1370] starting ApplyRowOperation cycle
0927 19:39:18.694489 (+ 327us) tablet.cc:1381] finished ApplyRowOperation cycle
0927 19:39:18.694758 (+ 269us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0927 19:39:18.694845 (+ 87us) write_op.cc:312] APPLY: finished
0927 19:39:18.695145 (+ 300us) log.cc:842] Serialized 23 byte log entry
0927 19:39:18.695299 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:18.695383 (+ 84us) write_op.cc:454] Released schema lock
0927 19:39:18.695460 (+ 77us) write_op.cc:341] FINISH: Updating metrics
0927 19:39:18.695750 (+ 290us) write_op.cc:489] Releasing partition, row and schema locks
0927 19:39:18.695776 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":137461,"apply.run_cpu_time_us":2304,"apply.run_wall_time_us":6211,"num_ops":1,"prepare.queue_time_us":89993,"prepare.run_cpu_time_us":16518,"prepare.run_wall_time_us":190682,"raft.queue_time_us":257361,"raft.run_cpu_time_us":22376,"raft.run_wall_time_us":75537,"replication_time_us":569977,"thread_start_us":465951,"threads_started":4,"wal-append.queue_time_us":262285}]]}
I20260927 19:39:18.689682 5608 raft_consensus.cc:482] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d1bc2cd9efe43399ddba396f5ddff85)
I20260927 19:39:18.728334 5608 raft_consensus.cc:3037] T 303e2824ebc54b4c9bce740f15e0105d P 69f09866d0a1477585f05b2f82ff4c67 [term 4 FOLLOWER]: Advancing to term 5
I20260927 19:39:18.728468 1991 raft_consensus.cc:1226] T c0017d74ca734f828f8096d7d474a209 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 13 FOLLOWER]: Rejecting Update request from peer 69f09866d0a1477585f05b2f82ff4c67 for earlier term 12. Current term is 13. Ops: []
I20260927 19:39:18.681100 5609 raft_consensus.cc:2726] T 0505ba7a2573409cbd518d77ccbcd201 P 3d1bc2cd9efe43399ddba396f5ddff85 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260927 19:39:18.706352 5611 consensus_queue.cc:1046] T 0505ba7a2573409cbd518d77ccbcd201 P 69f09866d0a1477585f05b2f82ff4c67 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d1bc2cd9efe43399ddba396f5ddff85" member_type: VOTER last_known_addr { host: "127.0.204.129"